This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
On the plausibility of Mars Bases vs that of AI
Responding to @FeepingCreature from last week:
That was a continuation of this discussion in which I say of AI 2027:
As to Mars:
Most of what I know here comes from reading Zach Wiener-Smith (of SMBC)'s A City on Mars. It was wildly pessimistic. For a taste, see Gemini chapter summaries and an answer to:
"Given an enormous budget (10% of global GDP) and current tech, how realistic is a 1 year duration mars base? an indefinite one? what about with highly plausible 2035 tech?"
I agree with the basic take there, both as a summary of the book and as a reflection of my broader (but poorly researched) understanding/intuition of the area: Mars is not practical. We could probably do the 1 year base if we don't mind serious risk of killing the astronauts (which, politically, probably rules it out. Maybe Musk will offer it as a Voluntary Exit Program for soon-to-be-ex X SWEs?)
My main interesting/controversial (?) take: there is an important sense in which Mars bases are much less of baseless scifi nonsense than AI 2027.
Mars is a question of logistics: on the one hand, building a self-contained, O2 recycling, radiation hardened, etc, base requires tech we may (?) not quite have yet. On the other hand, it strikes me as closer to refinements of existing tech than to entirely new concepts. Note that "enormous budget" is doing a lot of work in here. I am not saying it is practical to expect we will pay to ship all of this to Mars, or risk the lives, just that there is good reason to believe we could.
AI is a question of fundamental possibility: by contrast, with AI, there is no good reason to think we can create AI sufficient to replace OpenAI-grade researchers with forseeable timelines/tech. Junior SWEs, maybe, but it's not even clear they're on average positive-value beyond the investment in their future (see my previous rant about firing one of ours).
I don't understand how anyone can in good faith believe that even with an arbitrary amount of effort and funding, AGI, let alone ASI, is coming in the next few years. Any projection out decades is almost definitionally in the realm of speculative science-fiction here. Even mundane tech can't be predicted decades out, and AI has higher ceilings/variance than most things.
And yet, I am sensitive to my use of the phrase "I don't understand." People often unwittingly use it intending to mean "I am sure I understand." For example: "I don't understand how $OTHER_PARTY can think $THING." This is intended to convey "$OTHER_PARTY thinks $THING because they are evil/nazis/stupid/brainwashed." But, the truth of their cognitive state is closer to the literal usage: they do not understand.
So, in largely the literal sense of the phrase: I do not understand the belief in and fear of AI progress I see around me, in people I largely respect on both politics and engineering.
I put in a few thousand lines of code into an AI, ask it to change it to fix this issue... and it can do it. Sometimes it can't and it gives me a broken fix, other times I have to try several times or go through various stages of brainstorming, log-analysis, trial and error, workarounds.
If you're doing intellectual labour (this is intellectual labour, it's producing code that earns revenue) then you must be intelligent not in the 'answers toy questions' sense but the 'does useful work' sense. If it is intelligent, then AGI shouldn't be far away. It's only a matter of investment and incremental development.
Fundamental possibility is deader than disco. There is no reason for doubt at this point. You think there's an insurmountable gap between Junior SWE and senior SWE? That's ridiculously silly. There was no insurmountable gap between 'can't walk except on perfectly straight floor at 3 kph' and 'diving through hills, scrabbling around obstacles, getting up after being knocked over'.
There wasn't an insurmountable gap between 'the most deranged and hilariously stupid pretend harry potter writing imaginable' https://youtube.com/watch?v=6rEkKWXCcR4 and 'any story imaginable written in perfect, fully meaningful English albeit usually (but not always) lacking in literary merit but at a very reasonable price'.
There wasn't an insurmountable gap between 'can't identify a cat' and 'short videos of catgirls'.
There wasn't an insurmountable gap between 'literally no code at all' and 'Stackexchange withering away'.
There's no reason for doubt, you can't even give a reason except these elaborate statements of surety. I don't understand how what you're saying even resembles a valid argument. Forget about whether the premises are true or if the argument follows, there are no premises in what you're saying!
If an insurmountable gap there is, I claim is is not in the ability to do useful work, but in the ability to tell whether that work is useful or not.
Can you help me understand this claim more concretely? E.g. if an LLM had just successfully designed a bridge for me, but then I modified the design to make it not useful in some way, for some kinds of changes it wouldn't be able to tell if my change was good or not? But a human would?
Let's say you overlooked telling it about some fairly critical detail (your bridge is affected by gravity in some unusual way, or something, I'm not a civil engineer). It's not going to be able to figure that out on its own. And the world is full of such critical details that aren't captured in public datasets (or worse, are captured totally wrong). It will then confidently work off the wrong track, sometimes in subtle enough ways that it requires an expert to notice.
You can observe this right now if you work in a specialized or cutting edge field where solutions to problems are unintuitive. LLMs become worse than useless the more what you're trying to do works like that. And that's fine, it's just not the right tool.
But if there's something that looks like a fundamental limit of the approach vis à vis intelligence, that's what it looks like to me.
There's also more general issues with agentic systems specifically and how quickly they seem to fall victim to noise and hallucinations without human supervision, but I'm more on the fence as to whether this can be ameliorated.
Right, but neither would a human, unless they also had more direct access to the problem somehow. But that's what agentic scaffolding is for.
Even with tens of thousands of experts spending billions of dollars and R & D for a decade to solve these problems?
Seems like you're retreating to "I'm not sure"?
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link