This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
On the plausibility of Mars Bases vs that of AI
Responding to @FeepingCreature from last week:
That was a continuation of this discussion in which I say of AI 2027:
As to Mars:
Most of what I know here comes from reading Zach Wiener-Smith (of SMBC)'s A City on Mars. It was wildly pessimistic. For a taste, see Gemini chapter summaries and an answer to:
"Given an enormous budget (10% of global GDP) and current tech, how realistic is a 1 year duration mars base? an indefinite one? what about with highly plausible 2035 tech?"
I agree with the basic take there, both as a summary of the book and as a reflection of my broader (but poorly researched) understanding/intuition of the area: Mars is not practical. We could probably do the 1 year base if we don't mind serious risk of killing the astronauts (which, politically, probably rules it out. Maybe Musk will offer it as a Voluntary Exit Program for soon-to-be-ex X SWEs?)
My main interesting/controversial (?) take: there is an important sense in which Mars bases are much less of baseless scifi nonsense than AI 2027.
Mars is a question of logistics: on the one hand, building a self-contained, O2 recycling, radiation hardened, etc, base requires tech we may (?) not quite have yet. On the other hand, it strikes me as closer to refinements of existing tech than to entirely new concepts. Note that "enormous budget" is doing a lot of work in here. I am not saying it is practical to expect we will pay to ship all of this to Mars, or risk the lives, just that there is good reason to believe we could.
AI is a question of fundamental possibility: by contrast, with AI, there is no good reason to think we can create AI sufficient to replace OpenAI-grade researchers with forseeable timelines/tech. Junior SWEs, maybe, but it's not even clear they're on average positive-value beyond the investment in their future (see my previous rant about firing one of ours).
I don't understand how anyone can in good faith believe that even with an arbitrary amount of effort and funding, AGI, let alone ASI, is coming in the next few years. Any projection out decades is almost definitionally in the realm of speculative science-fiction here. Even mundane tech can't be predicted decades out, and AI has higher ceilings/variance than most things.
And yet, I am sensitive to my use of the phrase "I don't understand." People often unwittingly use it intending to mean "I am sure I understand." For example: "I don't understand how $OTHER_PARTY can think $THING." This is intended to convey "$OTHER_PARTY thinks $THING because they are evil/nazis/stupid/brainwashed." But, the truth of their cognitive state is closer to the literal usage: they do not understand.
So, in largely the literal sense of the phrase: I do not understand the belief in and fear of AI progress I see around me, in people I largely respect on both politics and engineering.
You're just asserting this without providing reasoning despite it being the entire crux of your post. I know it's not reasonable to expect you to prove a negative but you could have at least demonstrated some engagement with the arguments those of us who think it's very possible near term have put forward. You can at least put into some words why you think AI capabilities will plateau somewhere before openAI-grade researcher. How about we find out where we are relative to each other on some concrete claims and we can see where we disagree on them.
Do you agree that capabilities have progressed a lot in the last few years at a relatively stable and high pace?
Do you agree that it's blown past most of the predictions by skeptics, often repeatedly and shortly after the predictions have been made?
Are there even in principle reasons to believe it will plateau before surpassing human level abilities in most non-physical tasks?
Are there convincing signs that it's plateauing at all?
If it does plateau is there reason to believe at what ability level it will plateau?
I think if we agree on all of these then we should agree on whether to expect AI in the nearish term, I'm not committed to 2027 but I'd be surprised if things weren't already very strange by 2030.
Then it's good the 2027 claim isn't projecting out decades.
Yes and no. Clearly, things are better than even three years ago with the original release of ChatGPT. But, the economic and practical impact is unimpressive. If you subtract out the speculative investment parts, it's almost certainly negative economically.
And look - I love all things tech. I have been a raving enthusiastic nutjob about self-driving cars and VR and - yes - AI for a long time. But, for that very reason, I try to see soberly what actual impact it has. How am I living differently? Am I outsourcing much code or personal email or technical design work to AI? No. Are some friends writing nontrivial code with AI? They say so, and I bet it's somewhat true, but they're not earning more, or having more free time off, or learning more, or getting promoted.
Again, yes and no. Yes: Scott's bet about image generation. The ability to generate images is incredible! I would have never thought we'd get this far in my lifetime. No: anything sufficient to really transform the world. I have not seen evidence that illustrators etc are losing their jobs. I would not expect them to, any more than I would have from photoshop. See also Jevon's Pardox.
I think that is the crux of our disagreement: I hear you saying "AI does amazing things people thought it would not be able to do," which I agree with. This is not orthogonal from, but also not super related to my point: claims that AI progress will continue to drastically greater heights (AGI, ASI) are largely (but not entirely) baseless optimism.
Nothing has ever surpassed human level abilities. That gives me a strong prior against anything surpassing human level abilities. Granted, AI is better at SAT problems than many people, but that's not super shocking (Moravec's Paradox).
The number of people, in my techphillic and affluent social circle, willing to pay even $1 to use AI remains very low. It has been at a level I describe as "cool and impressive, but useless" forever. I will be surprised if it leaves that plateau. Granted, I am cheating by having a metric that looks like
x -> x < myNonDisprovableCutoff ? 0 : x, where x is whatever metric the AI community likes at any given point in time, and then pointing out that you're on a flat part of it.No, and that's exactly my point! AI 2027 says well surely it will plateau many doublings past where it is today. I say that's baseless speculation. Not impossible, just not a sober, well-founded prediction. I'll freely admit p > 0.1% that within a decade I'm saying "wow I sure was super wrong about the big picture. All hail our AI overlords." But at even odds, I'd love to take some bets.
Thanks for your thorough reply!
I think you're a little blinkered here. It takes more than a couple years to retool the whole economy with new tech. It was arguably a decade or more after arpanet before the internet started transforming life as we know it. LLMs are actually moving at a break neck pace in comparison. I work at a mega bank and just attended a town hall where every topic of discussion was about how important it is to implement LLM in every process. I'm personally working to integrate it into our department's workflow and every single person I work with now uses it every day. Even at this level of engagement it's going to be months to years cutting through the red tape and setting up pipelines before our analyst workflows can use the tech directly. There is definitely value in it and it's going to be integrated into everything people do going forward even if you can't have it all rolled out instantly. We have dozens of people whose whole job is to go through huge documents and extract information related to risk/taxes/legal/ect, key it in and then do analysis on whether these factors are in line with our other investments. LLMs, even if they don't progress one tiny bit further, will be transformative for this role and there are millions of roles like this throughout the economy.
Along with these amazing things it comes with a ripple of it getting steadily better at everything else. There's a real sense in which it's just getting better at everything. It started out decent at some areas of code, maybe it could write sql scripts ok but you'd need to double check it. Now it can handle any code snippet you throw at it and reliably solve bugs one shot on files with fewer than a thousand lines. The trajectory is quick and the tooling around it is improving at a rate that soon I expect to be able to just write a jira ticket and reasonably expect the code agent to solve the problem.
Certainly this is untrue. Calculators trivially surpass human capabilities in some ways. Nothing has surpassed humans in every single aspect. There is a box of things that AI can currently do better than most humans and a smaller box within that of things it can do better than all humans. These boxes are both steadily growing. Once something is inside that box it's inside it forever, humans will never retake the ground of best pdf scraper per unit of energy. Soon, if it's not already the case, humanity will never retake the ground of best sql script writer. If the scaffolding can be built and the problems made legible this box will expand and expand and expand. And as it expands you get further agglomeration effects. If it can just write sql scripts then it can just write sql scripts. If it's able to manage a server and can write sql scripts now it can create a sql server instance and actually build something. If it gains other capabilities these all compliment each other and bring out other emergent capabilities.
If people around you aren't paying for it then they're not getting the really cutting edge impressive features. The free models are way behind the paid versions.
AGI maybe not, but useless? You're absolutely wrong here. With zero advancement at all in capabilities or inference cost reductions what we have now, today, is going to change the world as much as the internet and smart phones. Unquestionably.
Come up with something testable and I am game.
What? That just isn't true. I've tried to have it write code and it's still in the same shitty place it was three years ago. You get something which looks correct, but maybe it is and maybe it isn't, and you have to double check every time. Which is to say, AI tools still slow you down rather than speed you up.
This is why I'm so skeptical that we'll have AI any time soon. The current tools aren't even good at the things their advocates say they are good at, let alone harder things. I have yet to see any substance behind the hype, at all.
Have you actually used the latest tooling? What tasks have you actually had it try? This seems incredibly unlikely to me.
I just use whatever ChatGPT has to offer, which would mean yes I'm using the latest tooling (since they keep it up to date). I've tried a variety of things - writing config files for programs we use at work, writing shell scripts, and asking it to explain how to do tasks in AWS CloudFormation. The first and the third tasks it just makes shit up (in some cases even dreaming up code which isn't even syntactically valid), I've found it to be completely useless for those. I've gotten some mileage in shell scripting, where it does fine as long as I keep the request small (like a few lines) so it can't trip over itself. But shell scripting is also an area I'm incredibly weak (essentially I can read bash but can't write it well at all), so it has the biggest gains to make over my own skill there. In cases where I actually know the language well, there's no benefit to me to use these tools. Like I said, if I have to check carefully every time I have it generate something (and you really do), then that's not actually speeding me up.
Here. I picked a random easyish task I could test. It's the only prompt I tried, and ChatGPT succeeded zero-shot. (Amusingly, though I used o3, you can see from the thought process that it considered this task too simple to even need to execute the script itself, and it was right.) The code's clear, well-commented, avoids duplication, and handles multiple error cases I didn't mention. A lot of interviewees I've encountered wouldn't do nearly this well - alas, a lot of coders are not good at coding.
Ball's in your court. Tell me what is wrong with my example, or that you can do this yourself in 8 seconds. When you say "like a few lines", is that some nonstandard usage of "few" that goes up to 100?
Even better, show us a (non-proprietary, ofc) example of a task where it just "makes shit up" and provides "syntactically invalid code". With LLMs, you can show your receipts! I'm actually genuinely curious, since I haven't caught ChatGPT hallucinating code since before 4o. I'd love to know under what circumstances it still does.
Wow! You asked chatgpt to solve a dead simple toy problem, and it solved it! I'm so impressed!!!! Surely this means that chatgpt is definitely capable of handling actual real world tasks.
More options
Context Copy link
I think that we are in the phase of chatgpt in which calligraphs and stenographs don't see the point of typewriters. I would definitely say that chatgpt has saved me a lot of time and made me more productive.
More options
Context Copy link
More options
Context Copy link
It's sensitive to context and prompting. When having it write bash scripts have you consider just dumping the man files into the context? Don't bother actually formatting them, just dump anything that could possibly be relevant into the prompt.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link