site banner
naraburns  nihil supernum  1yr ago (text post)   32915 thread views

Culture War Roundup for the week of February 27, 2023

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

10
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

I'm going to shamelessly steal @Scimitar's post from the Friday Fun thread because I think we need to talk about LLMs in a CW context:

A few months ago OpenAI dropped their API price, from $0.06/1000 tokens for their best model, to $0.02/1000 tokens. This week, the company released their ChatGPT API which uses their "gpt-3.5-turbo" model, apparently the best one yet, for the price of $0.002/1000 tokens. Yes, an order of magnitude cheaper. I don't quite understand the pricing, and OpenAI themselves say: "Because gpt-3.5-turbo performs at a similar capability to text-davinci-003 but at 10% the price per token, we recommend gpt-3.5-turbo for most use cases." In less than a year, the OpenAI models have not only improved, but become 30 times cheaper. What does this mean?

A human thinks at roughly 800 words per minute. We could debate this all day, but it won’t really effect the math. A word is about 1.33 tokens. This means that a human, working diligently 40 hour weeks for a year, fully engaged, could produce about: 52 * 40 * 60 * 800 * 1.33 = 132 million tokens per year of thought. This would cost $264 out of ChatGPT.

https://old.reddit.com/r/singularity/comments/11fn0td/the_implications_of_chatgpts_api_cost/

...or about $0.13 per hour. Yes technically it overlooks the fact that OpenAI charge for both input and output tokens, but this is still cheap and the line is trending downwards.

Full time minimum wage is ~$20k/year. GPT-3.5-turbo is 100x cheaper and vastly outperforms the average minimum wage worker at certain tasks. I dunno, this just feels crazy. And no, I wont apologize for AI posting. It is simply the most interesting thing happening right now.


I strongly agree with @Scimitar, this is the most interesting thing happening right now. If you haven't been following AI/LLM progress the last month, it has been blazingly fast. I've spent a lot of time in AI doomer circles so I have had a layer of cynicism around people talking about the Singularity, but I'll be damned if I'm not started to feel a bit uncomfortable that they may have been right.

The CW implications seem endless - low skill jobs will be automated, but which tribe first? Will HR admins who spend all day writing two emails be the first to go? Fast food cashiers who are already on their way out through self ordering consoles?

Which jobs will be the last to go? The last-mile problem seems pretty bad for legal and medical professionals (i.e. if an LLM makes up an answer it could be very bad) but theoretically we could use them to generate copy or ideas then go through a final check by a professional.

Outside of employment, what will this do to human relations? I've already seen some (admittedly highly autistic) people online saying that talking to ChatGPT is more satisfying than talking to humans. Will the NEET apocalypse turn into overdrive? Will the next generation even interact with other humans, or will people become individualized entirely and surround themselves with digital avatars?

Perhaps I'm being a bit too optimistic on the acceleration, but I can't help but feel that we are truly on the cusp of a massive realignment of technology and society. What are your thoughts on AI?

but I'll be damned if I'm not started to feel a bit uncomfortable that they may have been right.

Imagine how they feel.

The doomsday prophet spends large amounts of time warning that the end is nigh, and then the very portents he expected and claimed would occur... start occurring in a pretty unambiguous fashion.

I genuinely worry for the psyches of some very smart people who have made themselves particularly vulnerable to anxiety over AI acceleration.

I am slightly more worried about the psyches of some very smart people who are seeing all the same arguments and evidence for AI ruin and enthusiastically saying "hit the gas!"

Do you want your ability to think, reason, express yourself, generate insights and connections, to be owned and metered by OpenAI?

It’s hard to get the pro-AI crowd to think critically about these issues, because the assumption that the singularity is near is so prevalent.

“Do you think it’s a good thing for people to outsource the act of thinking itself to SV tech giants?”

“Well the AI is about to become literally God - could happen within the next few days, actually! - so it doesn’t really matter what I think or want. You can’t fight God, so all we can do is prepare for God’s arrival.”

You can’t have a nuanced discussion about social problems with people who think that the very notion of a “problem” itself is about to be dissolved.

Its also hard to take complaints about it seriously because tech has been 'replacing' various human tasks for over a century now, and to become a hardcore luddite only when it impacts writers, vs. all the various laborers and 'lower status' work that has been impacted, seems like a bad-faith special pleading.

GPT-3.5-turbo is 100x cheaper and vastly outperforms the average minimum wage worker at certain tasks.

Can you give some examples?

Like I've indicated, I work in a field - translation - where machine learning applications have already been commonplace for over a decade, seeing the development of machine translation from substandard early Google Translate effort that many people still associate with machine translation with the sophisticated effort put in by DeepL and other such engines. (ChatGPT also produces an OK effort, but as recounted here, still makes some more obvious mistakes than DeepL, at least when translating to Finnish.)

Furthermore, translation is a classic example of a field where overtly enthusiastic tech types have been predicting "human worker replacement by machine" for ages and ages now. An anecdote I often recount is when, during student days, I had been drinking with a friend who also studied translation and we went to a pizza place at the end of the night. We were accosted by a drunken engineer who started explaining how translation is a dead-end field and machines are going to replace translators any day now. I replied that when there's going to be a machine to replace the translators, it's also going to replace the engineers, and he got quiet and left. (At this moment, I wouldn't be surprised if the engineers weren't replaced before the translators.)

How has machine learning affected my workload? Well, the amount of money I've been making has if anything increased during the recent years, though this is also probably natural career development (and also a necessity to answer the rising inflation). For a long time, fair amount of my work has been checking and editing machine translation, a job that is common enough to just generally be referred by an acronym in the field (MTPE, Machine Translation Post-Editing). At the same time, I'm still able to charge 2/3 of my usual word-based rate for MTPE, evidence that there's still a lot of work to be done to not only fix the various errors that even advanced models make but get the "smell of machine translation" off the text.

Obviously, this is an issue of margins. While a lot of text is translated from one language to another, vastly greater amounts of texts getting produced right now aren't, including in commercial applications. Translator workloads getting lighter and translation getting cheaper has thus far just meant that more and more texts get translated now than previously. We'll see if the wall hits at some point.

Like I've also recounted previously, during the past few years I have, if anything, got less MTPE than previously, even though machine translation has improved. This is partly just random chance (ie. I've worked in projects which just aren't that suitable for modern MT applications) but also because some customers explicitly forbid translators from using MT, presumably because end-client companies are afraid that some trade secrets end up in Google's files. Of course I can't be sure they have means to actually check if I use MT if I just edit it good enough, but I can't not be sure of that, either, and getting caught for something like this would be a good way to lose a regular client.

From what I've learned, the biggest game changer in the field - from the point of view of a working translator - was when electronic communication enabled the translation from regular in-company jobs to enterpreneuer-based freelancing. This was actually going on while I was at the university - the teachers still mentioned in-company jobs as something to strive for, but often acknowledged they probably wouldn't be forthcoming and this would (negatively) affect pay for translators. Furthermore, even before machine translation, as such, got common, there have been the so-called translation memory programs, fairly simple tools that mostly replicate existing translations to new ones, and have done their share in making translation faster. Even in white-collar fields, the automation of gruntwork is hardly a new concept.

Of course we'll have to see in the coming years how, exactly, LLM's affect this field. One particular potential field for advancement would be when we get models that can, with some reliability, provess image, audio and text at the same time, since this might have a considerable effect on subtitling (or dubbing, but that is pretty rare in Finland, outside of children's programs). On the other hand, I work in a co-operative office and regularly chat with another translator who basically does not do MTPE at all, rarely uses MT in general and is generally not particularly aware of the developments in the machine-learning field. She seems to get by quite fine, nevertheless.

I edit highly technical documents, fixing grammar and spelling, and I've long thought a bot would eventually do my job. One of the companies I work for even uses a bot, sends us the bot-fixed papers, and then we double-check the bot, fixing its mistakes. Despite this going on for years, they are no closer to replacing us, in part because the standards keep getting raised. Like a recent update to the style guide suggests we should be using wherein in appropriate spots. Using where to mean in which has been normal in English for like forever, but it seems now that the bots are doing the easy stuff, we're expected to do the harder stuff too.

I've spent only a little time with ChatGPT and I've stated earlier that it is prone to unforced errors. But one of the bigger problems that I found is that it is prone to believing common falsehoods or myths. Go ask about the wage gap between men and women, which is just a bunch of statistical trickery but still a something that many people believe and consequently encoded into GPT.

Case in point during the week a bunch of Hacker News commentators took personal offence by a guy making the case that computer code should be written for computers if you want any kind of performance out of it. It is common opinion that code should only be written for other humans and writing it for computers is almost always a waste of time. It is the most prevalent attitude within my chosen profession and after 20 years I know that attitude of not writing code for machines is wasting performance. Guess what gets encoded into something that you cant reason with even less that a person that is convinced of superiority of his opinion? I've tried the output of GitHubs CoPilot, it does so many things wrong because the input to the models are wrong and incorrect code is so common. The ancient computer programmer adage Garbage In Garbage Out still holds true, and AI doesn't change that.

LLMs cannot improve from self-play. Once we get that, I don't know what will happen, might be direct-to-singularity, might not, but that issue shouldn't be a problem anymore.

ChatGPT won't write trash Python when it's had a million years of experience with performance tests.

I'm not sure exactly what 'self-play' means. Some papers found improvements by training LLMs on their own chain-of-thought outputs, and others, where you're trying to find something like a math proof or a program, train models to generate output, and use the successful outputs as more training material. The latter feel like 'a million years of experience with performance tests', but still write trash python often.

I wouldn't expect a paper where LLMs were trained on performance of their own generated code, and maybe fed profiler results, to report that afterwards, the LLMs still wrote trash python. Part of the issue here is that the LLMs cannot seek out problems to resolve on their own; though we should maybe expect such breakthroughs to only happen shortly before the singularity.

The problem we are looking here isn't doing selfplay for optimal code. The problem is to write something into a random adversarial environment. AI dominates Chess and Go with clear rules and perfect information that has trained through self play, but for Poker the results aren't as clear cut. All of that because of randomness and hidden information. So putting code into a distributed system within an organization full of internal corporate politics where a manager somewhere wants to sabotage and also there are external advesaries that want to mess with your system. Sure it can write optimal code for your computer through selfplay but actually delivering something to an enterprise setting that is a different ballgame, it is Chess vs Poker.

And even with perfect formal rules AI can still be tricked https://arstechnica.com/information-technology/2022/11/new-go-playing-trick-defeats-world-class-go-ai-but-loses-to-human-amateurs/

"Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far, the Universe is winning."

--Rich Cook

I don't think that has changed...

I'm just saying that inasmuch as LLMs are weak specifically at targeting objective metrics like performance, self-play should improve it. I'm not saying self-play is the panacea that'll give us AI, just that it will fill a hole in the existing methods.

I don't think that we are disagreeing at all here I'm just pointing out that having a target for self-play is going to be difficult. Because there are multiple dimensions to the problem of "not writing trash code" as it depends on whether or not it needs a theory of mind of actual people. Needing a theory of mind precludes self-play, that is always going to require input data.

ChatGPT's words are not even close to equivalent to a human's words. You have peek under the hood a little bit to understand why. ChatGPT is a prediction engine that predicts the next word in a sequence (as would be typical in its training corpus), and then applies that capability over and over again. ChatGPT has zero capability to abstract and apply its reasoning to its own thought process. ChatGPT can't wait and think about a question for a while before it starts answering.

The LLMs will continue to get better as researchers throw more parameters at the problem, but this avenue is ultimately a dead end for pursing general intelligence. ChatGPT is a neat parlor trick, but it can only make impressive-looking tech demos so long as the context is kept very narrow. Play around with it a little, and the cracks start to show.

All this is not to detract from your main thesis. Artificial general intelligence is still coming for lots of jobs at some unknown point in the future, but don't confuse ChatGPT with the herald of the jobs-apocalypse.

What, precisely, does "abstract" mean? If it means "do things like - generalize things to new situations, understand symbolic substitutions, form and use general principles" - well, look at this (screenshots are in english). It sure seems to have learned a general sense of how programming languages work. How is this 'zero capability to abstract'? For that matter, aren't induction heads effectively 'learning an abstraction', albeit an incredibly simple one?

Apologies for the abrasive tone: I take issue with your method and authorities, but would prefer you not to take it as a personal attack.

Funny that you talk about peeking under the hood, but later refer to Hofstadter. I'm sick of hearing about this guy – and cringe at the whole little adoring culture that nerds and «hackers» have built around his books. GEB and strange loops, this intellectual isomorphism of autofellatio, self-referential kabbalistic speculation detached from all non-synthetic evidence and loudly, proudly spinning its wheels in the air. «Dude, imagine thinking about… thinking! Isn't that, like, what programmers do? Woah…» It is pretty sad when a decently smart brain belongs to a man who happens to build a brand out of a single shower thought and gets locked by incentives into inflating it endlessly! Even worse when others mistake that for an epiphany, generations of poorly socialized kids looking for the promised deeper meaning, and establishment journalists respectfully asking the matured stoner for his Expert Input. (Then again, I may simply be envious).

Last I've seen Doug speak of machine learning, it was July 2022, and he was smug because he tricked GPT-3 into hallucinations (The Economist) :

I would call gpt-3’s answers not just clueless but cluelessly clueless, meaning that gpt-3 has no idea that it has no idea about what it is saying. There are no concepts behind the gpt-3 scenes; rather, there’s just an unimaginably huge amount of absorbed text upon which it draws to produce answers. But since it had no input text about, say, dropping things onto the Andromeda galaxy (an idea that clearly makes no sense), the system just starts babbling randomly—but it has no sense that its random babbling is random babbling. Much the same could be said for how it reacts to the absurd notion of transporting Egypt (for the second time) across the Golden Gate Bridge, or the idea of mile-high vases.

This is not to say that a combination of neural-net architectures that involve visual and auditory perception, physical actions in the world, language and so forth, might not eventually be able to formulate genuinely flexible concepts and recognise absurd inputs for what they are. But that still wouldn’t amount to consciousness. For consciousness to emerge would require that the system come to know itself, in the sense of being very familiar with its own behaviour, its own predilections, its own strengths, its own weaknesses and more. It would require the system to know itself as well as you or I know ourselves. That’s what I’ve called a “strange loop” in the past, and it’s still a long way off.

How far off? I don’t know. My record for predicting the future isn’t particularly impressive, so I wouldn’t care to go out on a limb. We’re at least decades away from such a stage, perhaps more. But please don’t hold me to this, since the world is changing faster than I ever expected it to.

Narrator's voice: «they were 4 months away from ChatGPT». Today, pretty much the same text-only GPT-3, just finetuned in a kinda clever way for chat mode, can not only recognize absurd inputs, but also explain the difference between itself and the previous version, better than Hofstadter can understand. This was done on top of the previous InstructGPT tuning, also misrepresented by Experts On AI with what is basically a tech-illiterate boomer's conspiracy theory:

Artificial intelligence is an oxymoron. Despite all the incredible things computers can do, they are still not intelligent in any meaningful sense of the word.

InstructGPT is then further fine-tuned on a dataset labeled by human labelers. The labelers comprise a team of about 40 contractors whom we hired through Upwork and ScaleAI.

OpenAI evidently employs 40 humans to clean up GPT-3’s answers manually because GPT-3 does not know anything about the real world.

I told one of my sons that the hand labelers would probably fix these glitches soon. Sure enough, I tried the same questions the next day, March 19, and found that the answers had indeed been cleaned up: ….

Gary: Can I use random numbers to predict presidential elections?

GPT-3, March 18: There is no definitive answer to this question. It depends on a variety of factors, including the accuracy of the polling data and the margin of error.

GPT-3, March 19: No, you cannot use random numbers to predict presidential elections.

The labelers will probably clean this up by tomorrow, but it doesn’t change the fact that so-called AI algorithms still do not understand the world and consequently cannot be relied upon for sensible predictions or advice.

Today we know that LLMs have what amounts to concepts. Today, people have forgotten what they had expected of the future, and this Sci-Fi reality feels to them like business as usual. It is not.

Every little hiccup of AI, from hallucinations to poor arithmetic, its critics put into the spotlight and explain by there not being any real intelligence under the hood, the sort they have. The obvious intellectual capacity of LLMs demonstrated by e.g. in-context learning is handwaved away as triviality. Now, like Boretti says, «frames or symbols or logic or some other sad abstraction completely absent from real brains» – now implementing that would be a «big theoretical breakthrough». We don't know if any of that exists in minds in any substantial non-metaphorical sense, or is even rigorously imaginable, but some wordcels made nice careers out of pontificating on those subjects. Naturally, if they can be formalized, it wouldn't be much of an engineering task to add them to current ML – the problem is, such reification of schemes only makes things worse. The actual conceptual repertoire developed by humble engineers and researchers over decades of their quest is much more elegant and expressive, and more deserving of attention today.

Do you really think that the idea of «predict next word in a sentence» provides sufficient insight about the under-the-hood intelligence of LLMs, when it is trivial to change the training objective to blank-filling, and RLHF guarantees that there exists no real dataset for which the «predicted» – or actually, chosen – token is in fact the most likely one in that context?

Or that the process of self-attention , actually undergirding those «predictions», is not «reasoning about reasoning» (because it cannot attend to… itself, the way you can attend to patterns of neuron spikes, presumably?

Or that recurrence is hard to tack onto transformers? (or for that matter specialized cognitive tools, multimodality etc.?)

And so on and so forth. But ultimately after years and years of falsified forecasts and blatant displays of ignorance by skeptics, it is time to ask oneself: isn't this parlor trick of a stochastic parrot impressive enough to deserve more respect, at least, than gimmicks of our fraudulent public intellectuals? It does, after all, make more sense when talking, and is clearly more able to grapple with new evidence.

@2rafa reasonably observes that human intelligence may be not so different from next word prediction. Indeed, if I close this tab (in Obsidian) and return to it in half an hour, I may forget where I was going with this rant and start with the most likely next word; and struggling for words when in an intense conversation is an easy way to see how their statistical probabilities affect reasoning (maybe I'm projecting my meta-awareness, lol). But even if the substrate is incompatible: why do we think our one is better? Why do we think it supports «real reasoning» in a way that mutiplying matrices, estimating token likelihood, or any other level of abstraction for LLM internals does not?

It is not obvious that the human brain is anywhere near optimal for producing intelligent writing, or for much of anything except being itself. We didn't evolve to be general-purpose thinkers, we are just monkeys who had our brains scaled up under selective pressures in some limited range of environments, with stupid hacks like the phonological loop and obsession with agency. Obviously LLMs are not optimal either (we are only pursuing them out of convenience), but they might still be better at producing our own text.

A plane doesn't flap its wings, but it definitely flies – and even though it's less efficient per unit of mass, in the absolute sense it does something no bird could. We do not understand birds well enough to replicate them from scratch, nor do we need to. Birds could never achieve enough for a truly general flight, «move anything across the Earth» kind.

We, too, aren't enough for truly general intelligence. It remains to be shown that LLMs aren't better fit for that purpose.

ChatGPT is a prediction engine that predicts the next word in a sequence ... . ChatGPT has zero capability to abstract and apply its reasoning to its own thought process.

Well, maybe. But even if it is "just" predicting the next word in the sequence, it turns out that "just" being able to predict word sequences endows you with some surprising capabilities, like being able to explain jokes faster and more accurately than some humans can.

I've always found discussions about whether AI can "really" think or "really" understand things to be rather boring. These questions are of philosophical interest only, but even as far as philosophical questions go, I've never found them to be particularly urgent. Some philosophical problems do keep me up at night; Searle's Chinese room is not one of them.

The only thing that really matters is the demonstrated capabilities of the AI, insofar as they can be empirically verified. When the robot comes for your job, it's of little comfort to remind yourself that it can't "really" think. What actually matters is that it took your job.

Whether AIs "really think" wasn't really the topic under discussion. Dijkstra compared that question to whether submarines swim, and I'm content with his answer.

I was more referring to the fact that while ChatGPT can do some cute stuff, it's not commercially relevant for any application that can't tolerate a 3% fly-off-the-rails rate, and increasing the model size won't address the architectural limitations.

Why are you so certain that human intelligence (at least in bulk) is not "predicting the next word in a sequence"?

Because we reason about concepts. Predicting the next word in a sequence is only the output filter that attempts to make said sequence intelligible to our interlocutor so that she may also reason about concepts in similar fashion. We're not perfect at it, but we're oriented toward something different.

It's telling that we'd have to tack on a calculator module to get ChatGPT to be able to do arithmetic reliably. There are probably a lot more less well-defined tasks, no more complicated than arithmetic, that ChatGPT can't do on its own, but the arithmetic is just the most glaring to see when it gets wrong.

My certainty is more of a gut feeling informed by Hofstadter's Gödel, Escher, Bach and the connection between strange loops and cognition. You're of course right that if you can predict the next word in a sequence well enough, you can do any intellectual task, including human cognition. But "well enough" can be a stand-in for arbitrary amounts of computation, and the transformer models don't do the necessary work. In particular, they're not reasoning about their reasoning faculties, which I believe is a key component to any general intelligence. And more parameters isn't going to get us there. We're at least one more big theoretical breakthrough away from useful machines that reason.

It's telling that we'd have to tack on a calculator module to get ChatGPT to be able to do arithmetic reliably

This is comical when you remember that humans, too, fail to do arithmetic reliably unaided. Why don't you try to add two seven-digit numbers, without any 'intermediate steps'? Also, 'chain of thought prompts' enable models to do some quantitiave problems pretty well.

In particular, they're not reasoning about their reasoning faculties, which I believe is a key component to any general intelligence

How precisely is a hunter-gatherer, or an average person 'reasoning about reasoning faculties' in their day-to-day life? Hunter-gatherers were "general intelligence". Why is that necessary to ... reliably add numbers?

also: let's assume that's true. why can't a neural network just encode it's "reasoning faculties" in a sufficiently large sequence of bits and then 'reason about its reasoning faculties' within those bits, the same way you claim humans do? (And if 'reasoning about reasoning' is a rationalist (in the philosophical sense not the rat sense) on the specialness and non-materiality of the human soul, as sometimes is ... does that really work)?

this isn't to say arithmetic proves the two are the same, but these objections seem weird.

I think I broadly agree. As it stands today, many (most?) organisation are horribly inefficient. But what this does is raise the ceiling of maximum productivity. We might be entering a world of hyper-startups, a world where a single guy can write code, write marketing copy, produce art assets, provide customer support, etc, and create more value than a competitor deploying 1000x more resources.

And people like work, need it even. People spend thousands of hours completing rote, boring tasks in online video games that are essentially analogous to actual labor;

Then why don't they do boring tasks that pay? They may seem boring, but they must find them fun.

In France, for example, profitable companies are literally banned from laying off large numbers of workers.

Even unprofitable companies, they won’t let die. My uncle worked for a company that hasn’t made a profit in years. They got ‘bought’ repeatedly by the company that promises to lay off the least people. From what he hears from former colleagues, a 300-man workforce looks at their shoes all day because there are no clients left. For years. Why don’t they leave? Their salaries and, eventually, severance pay is guaranteed to be payed out by a state-level insurance fund, because obviously the company has no money. Plus they would miss out on the generous unemployment insurance if they leave of their own accord. They aren’t even having fun, they spend their days in a mixture of gloom and excruciating boredom, this is a pure societal loss.

Why would another company but an unprofitable company? By the way, maybe you're French, so I should tell you it's "paid".

They don't really buy, hence the quotation marks. It's for a symbolic euro, or if their savior plan includes investment of their own money, they'll ask for the original debt on the bankrupt company to be discounted, tax breaks, etc.

I had my first experience this week using ChatGPT for my job. I'm a web developer. I manage all of the web properties for a small non-profit.

They have an old wordpress site that is bloated with a lot of plugins. Its often impossible to get support from any plugin developer because they take one look at the mess that I'm managing and nope their way out of helping.

ChatGPT was actually helpful. It helped me diagnose the exact problem. Then it helped me locate an area to implement solutions. Then it gave me a solution. Then it told me how to test the solution. And then the solution failed to do anything. I went through all the steps a few times, and was able to realize it was giving me some bad code. I kept prompting it in different ways and asking for explanations of its code. Eventually I got to a valid solution.

Anyways it was weird as hell. I've worked with junior programmers underneath me before. And I'd be happy to have someone like ChatGPT as a junior programmer beneath me. I'd never recommend they advance past junior programmer. But they basically make for a super googler + semi-dumb code thief.

And I'm not saying that to be like "oh look how crap AI is". Its more like "shit, its too far gone".

I have two young daughters. They are by most standard metrics pretty smart well adjusted little kids. I can say with strong confidence that my wife and I are better at our jobs than the AI. (both our jobs involve a fair bit of text manipulation + talking to people). But in twenty years when my daughters are entering the workforce I don't really have much confidence that they will be better at a text manipulation job than the current generation of AIs.

Forget future advancements, just using the current level of AI will eventually crowd out a bunch of entry level text manipulation jobs.

I look at my daughters playing at night and I think of what world they might grow up in. Right now they love playing a make believe version of day care. They tuck the babies in for nap time, feed them, and then spend an inordinate amount of time giving them diaper changes (including reactions to poopy diapers "eww stinky", or "oh good just pee"). I can't help but think that "daycare professional" might be an oddly resilient career path in the future. Its not like anyone is gonna submit their kids for surveilance in a daycare setting to train up a set of AIs.

Hopefully we'll get to a time where they are wondering about what useful jobs they can have. I was a bit of AI apocalypse skeptic a few years ago. Most of my skepticism is gone.

My prediction is if one of your daughters is in daycare, she may be in charge of watching AIs who are watching the children.

20 years ago we couldn't imagine today even tho nothing has really changed ... Except literally the entirety of human interaction in the western world has shifted to a small device in your hand. It's why I'm shocked when some like Chuck Klosterman doesn't feel like much has changed in our culture in 20 years (not his exact point). It's that the way culture is produced, shared, and taken in has so completely changed that it feels like nothing has.

And when AIs are watching our children in two decades, it'll feel just like yesterday.

Many businesses don't even know how to use excel or adobe correctly. I work in one such business. There's just no interest in doing new things, everyone is paid by the hour. And apparently the quality of our products and user experience (this is relating to authors in the publishing industry) is well ahead of our peers, who are much larger and wealthier. They are presumably even less organized than we are. I suggested using AI-generated images for our more abstract book covers - but got shot down for a fake reason. Nobody wants change. This may well be different in other fields, notably tech. However, I suspect it is true for most sectors.

I subscribe to Yudkowsky's school of thought where self-driving cars will not be seen before the apocalypse because our regulatory institutions are so incompetent and slow. I think widespread white-collar job automation won't happen within our dwindling lifespans because society is not run by intelligent energetic, innovative thinkers. It's run by people with Essence of Baby-Boomer, people who just prefer doing things as they used to be done, people who don't comprehend that there could be a faster way to do things, even if they just look for the excel tool designed for their purpose. Our website doesn't load fairly often and looks like it was made in 2004, despite our failed attempt to modernize it. How hard is it to set up a modern website that lets people reliably order books? Not very hard in the universal sense of human capacity, very hard for us.

If there is any risk of unemployment, I imagine many jurisdictions will manipulate requirements and definitions to prevent machines being used to replace politically influential constituencies. HR doesn't do much functional work today but they have political power, power to influence people. It's easiest to stop new things from happening, to use delaying tactics or raise objections. See the OSS's wonderful guide to institutional sabotage, which seems to have been widely adopted: https://twitter.com/CityBureaucrat/status/1450240118195986437

It seems so easy to raise some specious reason why AI couldn't be used for this job, or to cherrypick some failure and pounce upon it. People have never made egregious failures before! The EU for instance demands that all AI decisions be legible and explainable, which is basically impossible due to the fundamental nature of the technology.

I cannot imagine how this ends well for Western civilization. How can our governing institutions, which have failed all but one of their tests, manage to get this right? They failed on nuclear energy, on waging idiotic wars in the Middle East. They failed on China (publicizing our plan to lure them to liberal democracy via free trade and openness is like bluffing after showing your hand), they let China develop its industries when it was weak and challenge them now that they're strong, ceding the initiative completely. They failed on climate change - if it is a serious matter then we should've gone nuclear decades ago and since it isn't then we shouldn't be squandering trillions on renewables. Gain of Function - big fat failure. Not knowing for sure whether masks do anything to airborne pandemics - big fat failure.

The only thing our leaders managed to do right was not killing everyone with a nuclear war and that might well be a lucky chance given how irresponsible they were. How can they possibly get AI right? It's not a simple matter like 'not choosing to destroy the world in great power war'.

/images/1677911876674946.webp

That reminds me of one of my wife's previous employers. Everything about the product she managed was a mess, the development team was incompetent, lazy and rife with personal bickering. The code was the worst spaghetti shit ever created by mankind and the architect too incompetent and full of himself to be propose any revisions, even though it's was so bad that further development was practically impossible. The dedicated sales person was stupid and highly sensitive about it and regularly made enemies of friends and refused to do her work.

Senior managment was aware of the problems but was so anemic that it took them three years to even begin to act on it despite acknowledgeding the problems and believing them to be urgent. (Although when they did act they did actually take good steps to improve things).

I initially thought that surely this product could only be competitive due to some bizarre regulatory capture in a part of Sweden but no. She went to a sales conference with the most advanced and prestigious customer organisations in the US and they were amazed by the product and sold very lucrative contracts... God help us.

Want to guess the industry?

Tax software?

I cannot imagine how this ends well for Western civilization.

Are you implying that there is some other civilization that does things better?

Gain of Function - big fat failure.

This thing is probably the most blackpilling thing about today's world.

Who profits from GoF research? There is no big business or government interest, there are no massive profits, this thing goes on so few thousand people can publish peer reviewed articles no one will ever read and burnish their citation metrics.

Few thousand people with concrete names, faces and addresses.

And no one cares.

Not governments, not billionaires, not celebrities (to say nothing about general normie population). Not any "protestors" "activists" or "rebels". All people, including the most rich, powerful and influential ones who can nevertheless die together with billions of ordinary peons when the next oops happens.

And oopses like this are happening pretty much regularly.

Imagine a ship, old times sailing ship. Ship where midshipman one day decided to drill a hole below the waterline, just for fun, just becuase he could. And everyone - captain, officer, crew and passengers - just watches and says: not my problem.

If you need hard evidence that humanity as a whole deserves to die, this is it.

Are you implying that there is some other civilization that does things better?

Well the Chinese do some things better but other things worse.

There was another actual-officially-confirmed Covid lableak too, in Taiwan actually: https://www.rationaloptimist.com/blog/covid-lab-leak-in-taiwan/

I mirror your notion about deserving to survive. The universe we see is so enormously rich and vast. Just one star is worth so much for so long. Our whole civilization relies upon a sliver of 0.000000045% of the Sun's power that reaches us, which goes through all kinds of circuitous routes in plankton, fish, agriculture and coal before it gets to us. If it gets to us at all. Our Sun is peanuts compared to the O-type giants, nothing compared to the black hole at the centre of the galaxy or whatever comprises 'dark' matter. There's nigh-endless energy and resources available to us. If only we showed a little seriousness about capturing them, if only we considered things more carefully, if we organized ourselves more efficiently... I read Bostrom's Astronomical Waste and it's enormously moving. The stakes are so enormous and our effort to survive and prosper is so pathetic.

The last-mile problem seems pretty bad for legal and medical professionals (i.e. if an LLM makes up an answer it could be very bad) but theoretically we could use them to generate copy or ideas then go through a final check by a professional.

I predict absolutely nothing will happen to medical professionals because of AI. We've already had "AIs" (aka expert systems) that perform as well or better as trained medical professionals in diagnosis for decades, yet they're used approximately nowhere.

They don’t perform as well. Someone has to actually examine the patient, observe his state and put the findings into the expert system. The expert system cannot do that. What it can do, on the other hand, is relatively trivial for the doctor who does the examination.

I find that most people who think doctors (well, medical professionals) are easy to replace have a pretty limited understanding of what actually happens in healthcare. Sure if you occasionally have an ear infection or a sprained muscle that seems pretty easy and simple and replaceable. Even something like anesthesia, what this guy is just pushing some buttons right?

Well no.

You go into the hospital with trouble breathing, your doctor comes to see you. Your heart rate is elevated. Do you have a growing infection? Are you nervous talking to the doctor? Were you trying to work out because you have a date next week? Is this a side effect from the breathing medication we gave you? Were you just fucking your girlfriend? One of these requires immediate start of antibiotics, and patients can have more than one of them happening at the same time (and in my experience, have).

The algo is just going to start abx which is not harmless by any means. Decision support exists but it's uniformly terrible because it can't take into account the gestalt and patients usually have multiple things going wrong (both inpatient and outpatient). Young and healthy people with a single sick complaint is approximately zero percent of the work in healthcare but also 100% of what is replaceable with decisions support right now.

In a U.S. ED we have multiple layers of triage and knowledge running from triage nurses, to mid level providers, to ED physicians to IP docs and consultants. We know that the lower levels on this scale are inferior (and that includes ED physicians) because we observe it on a daily basis.

Current decision support tools can't even read an EKG, the amount of development required to deal with the messy complexity of people (including the fact that people will misinform you both intentionally and unintentionally) is immense and god help us if the people like Cim who think we aren't doing anything useful or important get their way.

Disagree somewhat.

You go into the hospital with trouble breathing, your doctor comes to see you. Your heart rate is elevated. Do you have a growing infection? Are you nervous talking to the doctor? Were you trying to work out because you have a date next week? Is this a side effect from the breathing medication we gave you? Were you just fucking your girlfriend? One of these requires immediate start of antibiotics, and patients can have more than one of them happening at the same time (and in my experience, have).

Yes, this gestalt reasoning, this gut feeling, "does the patient look sick?" is important, but significantly this is now a thing that machines can do, and are continuously improving at.

Current decision support tools can't even read an EKG

Yes the interpretations that are printed out on those machines are shit. But this is not state of the art. It is possible to do better

We develop an algorithm which exceeds the performance of board certified cardiologists in detecting a wide range of heart arrhythmias from electrocardiograms recorded with a single-lead wearable monitor.... We exceed the average cardiologist performance in both recall (sensitivity) and precision (positive predictive value).

I absolutely agree it will get there, and this update that we have the technology to read EKGs correctly now is profoundly unshocking, but the fact that it isn't in use is telling (healthcare has tons of barriers including the regulatory) and the ability to form a good gestalt is a very complicated and hard skill.

AI will come for us - motivation is high because of the cost (in dollars) but the level of care is high because of the cost (in lives). We'll probably be one of the later manifestations and docs won't get cut out.

This topic irks me so much because one of the strongest patterns in American healthcare is outside industries thinking it is easy, rolling in, failing miserably, and leaving behind a trial of broken lives (especially coming in from tech and finance). Winners do exist (see like PE in EM and HCA) but mostly do so through illegal activity and profoundly unethical behavior.

Thanks, this sounds more like how I was thinking about it. Like, maybe the algorithm can, or at least could, make okay decisions if it had all of the information. But then isn't actually gathering all of the information and getting it into a form that could be entered or written down somewhere like 80% of what doctors do anyways? I'm not sure if it matters how good the algorithm is if any professional could have already made the best practical decision before they even would have been able to enter all of the information into some system anyways.

So we have a ton of top of the line decision support tools right now, (including things like auto-read for EKGs, suggestions to put in antibiotic if the computer thinks someone is septic, etc.) the problem is that they suck and are intrusive and annoying. This is important, not only do they need to be more right but they also need to be consistently right - people are trained just to ignore them and if you go from being helpful from 5% of the time to 30% of the time they'll still be functionally useless. If we get to a 70% range situation people will ignore them out of habit and ingrained mistrust.

That problem aside...why is this shit so hard?

It's not because medicine is complicated (it is, but that's not the problem*), LLM are perfect for digging through a bunch of data and such. It's because people are complicated. People come in with a severe illness and complain about something else, ignore a diagnostically critical symptom, report pain in the "wrong" quadrant for the pathology (happens all the damn time).

The decision support tool needs to handle this ambiguity gracefully, have some mechanism for sussing out the correct shit from the patient, and have graceful way of handling the editorializing of whoever is recording and entering the data (and ideally in a timely fashion as you mention).

And then you have super significant but more arcane layers to the problem. Okay my patient has a kidney issue and a heart issue. My decision support tool can help and send me the most updated guidelines. Well where are we pulling from? Cards or Neph? One is shouting Blue and the other is shouting Yellow and depending on which Ivory Tower Institution you pull from the shades of those colors are going to be wildly different.

Research in medicine is difficult and fraught and ethically complicated and we don't have enough high quality recommendations to load this stuff with.

In Europe they manage appendicitis mostly medically, in the U.S. we operate. You ask a surgeon here why the difference and they'll probably say it's because we are fatter. Is that right? Fuck if I know, but we can't agree on the most simple of management.

*I have no idea why the EKG reads are bad, that's pretty damn simple and doesn't bode well for getting anything more complicated done.

The guy who presses his finger into your muscle and asks if it hurts before recording it doesn’t need a decade of training and a $300,000 salary.

Is this the GP everyone usually goes to first you mean, or hospitals? Because sure, you can have a minimum wage triage clerk taking pulse, heart rate, breathing and blood samples, and recording the data.

But then you need someone to interpret the findings. Is that muscle pain due to what? There's a lot of things it can be, and that's where the decade of training comes in. For a hospital, having the intake clerks do the grunt work while they then park you for six hours in the waiting room until the blood work comes through and The Consultant has had the chance to look at everything for five minutes before he decides to send you home is very doable. If you want to replace The Consultant with the AI, first I don't think it will do much to cut down on the six hours waiting time (organic chemical reactions go along at their own pace, plus however many tests are ranged up to be done) and second, that's fine right up until there's a mix-up in the blood work or the right test wasn't done and it turns out to be the sign of something more serious. That already happens in hospitals.

I'm not saying AI in medicine is a bad idea, I'm saying that we're likely to see it used first the way the insurance company triage phone lines are now used, and all it takes is one unfortunate death to set back the adoption of the technology. I could see it being adopted in hospitals, but for the local practitioner where people go first, you do need the level of training and expertise that can recognise "this is a muscle strain" from "this is something more serious, you need to go to the hospital right now".

agree. Robotic surgery hasn't put a dent in salaries either.

Not the same at all though. Robotic surgery enhances the power of the human surgeon, making more different types of surgery possible and thus increasing demand.

On the other hand, with near-term AI advancements, specialties like radiology could be completely eliminated. Radiologists wouldn't gain more power. They would be completely useless.

Come to think of it, are radiologists starting to sweat? IIRC, radiology is one of the hardest specialties to get a residence in. Is that starting to loosen?

I recall hearing about radiology in particular, how it's already been subject to being upset by telehealth and cheaper overseas competition. But radiologist is still a good living.

I'm not sure you understand what anesthesiologists actually do. Gas providers are pilots, if you assume 100% of the work is when the plane is on cruise control, sure... But take off and landing are a thing. And emergencies. And routine (and emergent) prep and post work. Also the other things gas does in the hospital (especially for OB and EM but also general procedural work).

This post on meddit outlines some specific.

https://old.reddit.com/r/medicine/comments/10wj8ma/anesthesiologists_how_well_could_crnas_do_your_job/j7ohdyo/

It depends on the region but the average is more 300-500.

And because you don't know anything about Anesthesia. To some extent this is fair, patients don't interact with gas much and medical TV shows are uniformly misinforming, but you haven't show any evidence for the idea that they don't do anything complicated or useful to be a reasoned and informed belief.

The pilot analogy is apt. Most people intuit that takeoff and landing are harder, and that emergencies sometimes happen. They don't know about ground prep and the other stuff pilots do. I'm not sure you are naturally making those connections here and "lol nah I'm not going to read that" does not help you become better informed.

To be maximally charitable to you it is reasonable to figure this guy is talking up what gas does (or more realistically is ignoring the difference between lazy docs and hard working ones) but that doesn't change the fact that their are multiple fundamental every shift job roles you aren't aware of.

I'd love to see an AI try and intubate someone, or deal with a difficult airway. Or give spinal anaesthesia, or an epidural, or a brachial plexus block. Anaesthesiology is much more procedure oriented than you think. There is no chance it could be done by a robot without tremendous technological advances.

Anaesthesiologists are similarly completely unnecessary

How? I imagine the person making sure you are anaesthetised, breathing, and not having a heart stoppage during surgery is a bit necessary? Or are you saying this is something that could be completely automated?

The AMA cartel will ensure Doctors get paid 400k a year even a century after the Singularity.

You do realize the AMA primarily lobbies in support of lower salaries for doctors right?

surely the AMA can't prevent these innovations from being used in other countries?

Would be the ultimate example of "America leads the way in new medical tech, the rest of the world leads the way in making it not break the banks of patients." Medical tourism would skyrocket.

Imagine craft guilds having more influence centuries ago, imagine laws ensuring that every newfangled machine had to be operated only by fully trained and qualified Master Spinner, Master Weaver and Master Tailor.

Your clothes would cost half of your yearly income, but they would be awesome.

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats