site banner
PaperclipPerfector  11mo ago (text post)   30543 thread views

Culture War Roundup for the week of July 3, 2023

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

6
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

I earnest disagree. If you check the GPT-4 white paper, the original base model clearly had a sense of internal calibration, and while that was mostly beaten out of it through RLHF, it's not entirely gone.

They have a genuine understanding of truth, or at least how likely something is to be true. If it didn't, then I don't know how on Earth it could answer several of the more knotty questions I've asked it.

It is not guaranteed to make truthful responses, but in my experience it makes errors because it simply can't do better, not because it exists in a perfectly agnostic state.

I think you are flatly wrong about this. I've tried to find literally anything to back up what you are saying, and come up with zilch. Instead, I wound up with this.

https://www.scribbr.com/ai-tools/is-chatgpt-trustworthy/

A good way to think about it is that when you ask ChatGPT to tell you about confirmation bias, it doesn’t think “What do I know about confirmation bias?” but rather “What do statements about confirmation bias normally look like?” Its answers are based more on patterns than on facts, and it usually can’t cite a source for a specific piece of information.

This is because the model doesn’t really “know” things—it just produces text based on the patterns it was trained on. It never deliberately lies, but it doesn’t have a clear understanding of what’s true and what’s false. In this case, because of the strangeness of the question, it doesn’t quite grasp what it’s being asked and ends up contradicting itself.

https://www.scoutcorpsllc.com/blog/2023/6/7/on-llms-thought-and-the-concept-of-truth

Thus far, we’re really just talking about sentence construction. LLMs don’t have a concept of these as “facts” that they map into language, but for examples like these - it doesn’t necessarily matter. They’re able to get these right most of the time - after all, what exactly are “inferences” and “context clues” but statistical likelihoods of what words would come next in a sequence?

The fact that there is no internal model of these facts, though, explains why they’re so easily tripped up by just a little bit of irrelevant context.

https://fia.umd.edu/comment-llms-truth-and-consistency-they-dont-have-any-idea/

They have zero idea what's true. They only know the probabilities of words in text. That's NOT the same thing as "knowing" something--it's a bit like knowing that "lion" is the most likely word following "king of the jungle..." without having any idea about monarchies, metaphor, or what a king really is all about.

The folks at Oxford Semantic Technologies wrote an interesting blog post about LLMs and finding verifiable facts. They call the fundamental problem the "Snow White Problem." The key idea is that LLMs don't really know what's true--they just know what's likely.

He is likely referring to this from pages 11-12 of the GPT whitepaper:

GPT-4 can also be confidently wrong in its predictions, not taking care to double-check work when it’s likely to make a mistake. Interestingly, the pre-trained model is highly calibrated (its predicted confidence in an answer generally matches the probability of being correct). However, after the post-training process, the calibration is reduced (Figure 8).

In any case, the articles you quote are oversimplified and inaccurate. Predicting text (and then satisfying RLHF) is how it was trained, but the way it evolved to best satisfy that training regime is a bunch of incomprehensible weights that clearly have some sort of general reasoning capability buried in there. You don't need to do statistical tests of its calibration to see that, because something that was truly just doing statistical prediction of text without having developed reasoning or a world-model to help with that task wouldn't be able to do even the most basic reasoning like this unless is already appeared in the text it was trained on.

It's like saying "humans can't reason, they're only maximizing the spread of their genes". Yes, if you aren't familiar with the behavior of LLMs/humans understanding what they evolved to do is important to understanding that behavior. It's better than naively assuming that they're just truth-generators. If you wanted to prove that humans don't reason you could point out all sorts of cognitive flaws and shortcuts with obvious evolutionary origins and say "look, it's just statistically approximating what causes more gene propagation". Humans will be scared of things like spiders even if they know they're harmless because they evolved to reproduce, not to reason perfectly, like a LLM failing at Idiot's Monty Hall because it evolved to predict text and similar text showed up a lot. (For that matter humans make errors based on pattern-matching ideas to something they're familiar with all the time, even without it being a deeply-buried instinct.) But the capability to reason is much more efficient than trying to memorize every situation that might come up, for both the tasks "predict text and satisfy RLHF" and "reproduce in the ancestral environment", and so they can do that too. They obviously can't reason at the level of a human, and I'd guess that getting there will involve designing something more complicated than just scaling up GPT-4, but they can reason.

You don't need to do statistical tests of its calibration to see that, because something that was truly just doing statistical prediction of text without having developed reasoning or a world-model to help with that task wouldn't be able to do even the most basic reasoning like this unless is already appeared in the text it was trained on.

I opened up Bing Chat, powered by GPT4, and I tried that example. I got "The diamond is still inside the thimble inside the coffee cup on the kitchen counter". In fact, I've yet to see a single example of an LLM's supposed ability to reason replicated outside of a screenshot.

Despite being based on GPT-4 Bing is apparently well-known for performing dramatically worse. There have been some complaints of GPT-4's performance degrading too, presumably due to some combination of OpenAI trying to make it cheaper to run (with model quantization?) and adding more fine-tuning trying to stop people from getting it to say offensive things, but hopefully not to the extent that it would consistently fail that sort of world-modeling. (If anyone with a subscription wants to also test older versions of GPT-4 it sounds like they're still accessible in Playground?)

I don't think it's plausible that all the examples of GPT-4 doing that sort of thing are faked, not when anyone shelling out the $20 can try it themselves. And people use it for things like programming, you can't do that without reasoning, just a less familiar form of reasoning than the example I gave.

I don't think it's plausible that all the examples of GPT-4 doing that sort of thing are faked, not when anyone shelling out the $20 can try it themselves. And people use it for things like programming, you can't do that without reasoning, just a less familiar form of reasoning than the example I gave.

My problem is, while I'm sure that not all the examples of GPT-4 seeming to get complex reasoning tasks are fake, if they cannot be replicated, what good are they? If GPT-4's ability to "reason" is ephemeral and seemingly random, is it really reasoning, or is it just occasionally getting lucky at ordering abstract tokens for it's monkey overlords?

You know, it's funny, I went through the linked whitepaper. Skimmed mostly. It made few positive, objective claims about GPT4's ability to reason. It mostly said it could reason "better" than previous iterations, and had been trained on a dataset to encourage mathematical reasoning. Notably they say:

It can sometimes make simple reasoning errors which do not seem to comport with competence across so many domains

I saw some the prompts where they asked GPT-4 to explain it's reasoning, and was underwhelmed. They were extremely rudimentary mathematical tasks of the 5th grade word problem sort, and it's purporting "reasoning" could have easily been imitating training. When I saw that, I took a closer look at how it performed in assorted test, and saw it comprehensively failed the AP English Language and Composition and AP English Language and Literature tests. Which makes sense to me, because a lot of those tests involve more generalized and flexible reasoning than the sorts of formalized mathematical logic examples it might plausibly be trained to imitate.

When I saw that, I took a closer look at how it performed in assorted test, and saw it comprehensively failed the AP English Language and Composition and AP English Language and Literature tests. Which makes sense to me, because a lot of those tests involve more generalized and flexible reasoning than the sorts of formalized mathematical logic examples it might plausibly be trained to imitate.

Come on, most of the UK parliament can't even give the probability of two coins both coming up heads: https://www.bbc.com/news/uk-19801666

Most people can't even imitate intelligence, by your logic.

GPT-4 has vastly superhuman knowledge, superhuman language knowledge, superhuman speed. Its reasoning skills are well above most of humanity. Most people can't program at all, let alone in all the languages, know how to use so much software like it can. These niggling flaws in AP English and Composition probably have more to do with the arcane and arbitrary scoring mechanism in those tests. It can write just fine. Its prose is not amazing and tends to be rather cliche and predictable, yet that has a lot to do with the RLHF.

Come on, most of the UK parliament can't even give the probability of two coins both coming up heads: https://www.bbc.com/news/uk-19801666

That is astonishing.

I wonder how well the average EU parliamentarian or US congressman etc. would do. I can’t imagine that the average Politburo member in China would be this bad.

Well, Britain has been in decline for the last century... I think we could learn a great deal by watching what they've done, what they do and committing to the opposite. They deliberately crushed Birmingham for instance - 'too much development in this rich industrial region, stop!' https://unherd.com/2020/09/the-plot-against-mercia/

And apparently NHS maternity incompetence costs twice as much as NHS maternities themselves: https://www.thetimes.co.uk/article/maternity-payouts-twice-cost-of-care-times-health-commission-svdhsjhqk

The Chinese politburo has a fair few with a basis in science and engineering, I made a post about it a while ago which was contested. Xi at least has an engineering background, other Politburo members have tiny wikipedia pages. https://www.themotte.org/post/238/what-if-your-entire-worldview-was/44213?context=8#context

The Chinese politburo has a fair few with a basis in science and engineering, I made a post about it a while ago which was contested. Xi at least has an engineering background, other Politburo members have tiny wikipedia pages. https://www.themotte.org/post/238/what-if-your-entire-worldview-was/44213?context=8#context

My understanding was that politburo members were, at one point, mostly or completely of an engineering or science background, but this has relaxed recently. Regardless, the point stands.

I think this holds for the other East Asian countries though, even with a pretty low estimation of the Diet or the National Assembly of Korea.

Wonder what it’s like in other countries.

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats