domain:mgautreau.substack.com
Primaprimaprima
I am trying so hard not to say the MBTI type of everyone I reply to
5hr ago
Training language models to be warm and empathetic makes them less reliable and more sycophantic:
Artificial intelligence (AI) developers are increasingly building language models with warm and empathetic personas that millions of people now use for advice, therapy, and companionship. Here, we show how this creates a significant trade-off: optimizing language models for warmth undermines their reliability, especially when users express vulnerability. We conducted controlled experiments on five language models of varying sizes and architectures, training them to produce warmer, more empathetic responses, then evaluating them on safety-critical tasks. Warm models showed substantially higher error rates (+10 to +30 percentage points) than their original counterparts, promoting conspiracy theories, providing incorrect factual information, and offering problematic medical advice. They were also significantly more likely to validate incorrect user beliefs, particularly when user messages expressed sadness. Importantly, these effects were consistent across different model architectures, and occurred despite preserved performance on standard benchmarks, revealing systematic risks that current evaluation practices may fail to detect. As human-like AI systems are deployed at an unprecedented scale, our findings indicate a need to rethink how we develop and oversee these systems that are reshaping human relationships and social interaction.
Assuming that the results reported in the paper are accurate and that they do generalize across model architectures with some regularity, it seems to me that there are two stances you can take regarding this phenomenon; you can either view it as an "easy problem" or a "hard problem":
-
The "easy problem" view: This is essentially just an artifact of the specific fine-tuning method that the authors used. It should not be an insurmountable task to come up with a training method that tells the LLM to maximize warmth and empathy, but without sacrificing honesty and rigor. Just tell the LLM to optimize for both and we'll be fine.
-
The "hard problem" view: This phenomenon is perhaps indicative of a more fundamental tradeoff in the design space of possible minds. Perhaps there is something intrinsic to the fact that, as a mind devotes more attention to "humane concerns" and "social reasoning", there tends to be a concomitant sacrifice of attention to matters of effectiveness and pure rigor. This is not to say that there are no minds that successfully optimize for both; only that they are noticeably more uncommon, relative to the total space of all possibilities. If this view is correct, it could be troublesome for alignment research. Beyond mere orthogonality, raw intellect and effectiveness (and most AI boosters want a hypothetical ASI to be highly effective at realizing its concrete visions in the external world) might actually be negatively correlated with empathy.
One HN comment on the paper read as follows:
A few months ago I asked GPT for a prompt to make it more truthful and logical. The prompt it came up with included the clause "never use friendly or encouraging language"
which is quite fascinating!
hydroacetylene
6hr ago
People been farming long enough that we’ve evolved some instincts. Little kids will plant random stuff in the dirt as a form of play. And theres just a satisfaction from eating your own homemade fruit that doesn’t come from store bought stuff, even the fancy organic kind. Likewise people seem to generally like animal husbandry.
Jiro
6hr ago
I was under the impression that the indigenous people were one of the problems with the current deal--they aren't from Mauritus and if you're going to return the islands to anyone, you return it to them, not to Mauritus, which the deal specifically didn't do. (And if you check a map, the islands are nowhere near Mauritus either.)
It's true that they aren't ancient, but they were expelled against their will, so they should still have some right to the islands.
07mk
6hr ago
I think what puts us between a horse and a hard place in this situation is that the default that women have chosen - something I certainly can't blame them for - is to be emancipated and then hope that someone else solves all their problems, and this combination of emancipation + hoping for a savior seems to result in poor life satisfaction, arguably even poorer than non being emancipated and then hoping that someone else solves all their problems. Whether this means that de-emancipating and hoping that someone else solves all their problems will have positive impacts is an open question. It's also arguable that being emancipated and using free will and agency to give away control to others and be unsatisfied about it is better in some way than to not be emancipated and while being forced into a life that's more satisfying. What I think most people would consider the golden path or the ideal outcome is women embracing their semi-recent emancipation and the agency and responsibility that goes along with it to solve their own problems, but recent history in sociopolitical movements relating to women's issues shouldn't give us much hope for that happening anytime soon. Hence why there appears to be no good option, just awful and more awful ones.
faceh
6hr ago
I think the subset of the human species that has the necessary skills to achieve interplanetary spaceflight is probably going to figure something out in time.
Whether that will be enough to keep the species as a whole at a post-industrial revolution tech level, though, I dunno.
Your point is scarily plausible, though.
hydroacetylene
6hr ago
Is there a way to have blood donated to use, personally?
hydroacetylene
6hr ago
Indigenous just means ‘the last non-white people in a region’.
hydroacetylene
6hr ago
an extremely outdated scientific theory that has connections to Theosophy.
‘Scientific’.
Was lemuria ever a science hypothesis? I was under the impression that even back in the day it was ancient aliens tier.
Apparently the US also has military infrastructure on the islands, and Trump was a fan of the deal?
I highly doubt Mauritius is dumb enough to tell the US to take their base away.
Corvos
6hr ago
Maybe, though I think that's a mistake on his part. In any case it sounds like I misread you:
I realise this doesn’t sound correct to you
so apologies for that.
DefinitelyAnAlt
6hr ago
Feeling positively overwhelmed at my new job.
My team is 'cracked' (as the kids call it) and I'm going in with a combination of excitement and nervousness. I've prided myself on being a clinger, hanging on for dear life. Today, I'm once again the stupidest guy in a room. I'm looking forward to it.
In my experience, surviving is an stronger motivator than thriving. I've jumped into the deep end of the pool, goal for the next 1 year is to survive. (Gotta hit that 1 year cliff)
shamgar
6hr ago
I think came across SSC when somebody mentioned it somewhere in a podcast in 2019. I was pretty hooked on the list of most reads blogs at the time and kept reading the new blogs from Scott for a bit and slowly made my way to the SSC subreddit en then to the Motte, it's been years since I actually frequently read the blog itself by now. I know a fair bit about rationalism through osmosis by reading a bit of SSC and lurking on the Motte for years, but I've never identified with the movement. To be honest, I never found any rationalist except Scott himself to be an interesting read. I checked out stuff like Eliezer Yudkowsky, Lesswrong, Gwern, etc. a couple of times when it was mentioned on SSC but I never understood the appeal.
hydroacetylene
6hr ago
Convert to Catholicism and take an uber-conservative priest as a spiritual director?
I’ve heard that courage international is similar to what I had but in a group setting.
hydroacetylene
6hr ago
Yes, we should plant trees. But can you blame women for not signing up to be de-emancipated in the hopes someone else solves all their problems? Putting the cart before the horse there.
FarmReadyElephants
6hr ago
Also, you are misunderstanding 13 52. 13 52 means 13% of the population does 52% of the murders. That is higher both on a per capita basis and on an absolute basis than any other group, by the necessity of percentage math.
shamgar
6hr ago
I think there is probably a correlation, but I reckon the causality might run the other way. Countries that are more inclined towards government overreach will see more tangible effects of wokeism, whereas in countries with less of a tendency towards government overreach even if wokeism becomes popular among the PMC or whatever, it will be more difficult to implement woke stuff in government policy.
The_Nybbler
If you win the rat race you're still a rat. But you're also still a winner.
6hr ago
The U.K. is indeed pro-American, but if Trump figures he can just act as if the U.S. owns the place and Mauritus can suck it, that's even better, at least from Trump's perspective.
If he thinks that (and I’m not sure he does, America has been very anti the sale) then he’s an idiot. The UK is reflexively pro-America.
I realise this doesn’t sound correct to you, because the UK criticises America (especially Trump and MAGA) so much, but it’s still true. The UK sees its criticism as coming from a colleague in the same tent, and will never side with China or Russia or really any other power on a matter of serious geopolitics. All we ask for in return is some subsidies and some head pats but we will make do even without.
There are maniac Leftists of course, even in the government, but they hold no influence on these matters and they will certainly not support Russia/China over America.
I doubt the same is true of the Mauritius.
I posted a while back about how practicing driving was tiring to me in a way few activities are. I'm happy to report that I now have my license and a car. Even though I live in the city so parking is less convenient, just feeling of being able to go anywhere without having to check public transport is very freeing.
The reason I only bothered so late in life is that I have been living in the city for all my adult life so it was never a necessity, and my province imposes taking expensive classes (around a 1000$) for one year before you're able to get your license, so the cost and delay killed my motivation to get it for recreational purposes. But now I'm thinking to move out of the city so having a license is a prerequisite.
It's still tiring and stressful to me in the kind of road environment I have practiced less in (highways, on-ramps, service roads) but busy city roads are getting to be second nature now.
The_Nybbler
If you win the rat race you're still a rat. But you're also still a winner.
6hr ago
Apparently the US also has military infrastructure on the islands, and Trump was a fan of the deal?
Trump probably figures the US can bully Mauritus much easier than it can bully the U.K. After all, there's little chance that Trump will be swayed by the scolding of some international court.
The_Nybbler
If you win the rat race you're still a rat. But you're also still a winner.
6hr ago
The treaty establishing the jurisdiction of the court in question specifically excluded the U.K.s relationships with its former possessions. A lawyer-brained person would have no problem ignoring the court under such circumstances.
WhiningCoil
Ghost of Quokka's Future
7hr ago
Yeah, but that's up there with milquetoast phrases like "Boys will be boys" IMHO.
Like, by comparison, it is not uncommon for women to check in with friends before and after dates "Just in case". It's just common knowledge that it's a thing you should do. Maybe mother's tell their daughters, I wouldn't be shocked. But I knew many women who had this sort of buddy system when they were going on dates. Along with dozens of other rules of thumb to protect themselves in case the man was a scumbag or violent.
What defenses are men armed with? "Don't stick your dick in crazy" I guess that works, but what is crazy? Well now we're right back to red-pillology as the only definer of women. And generally, after they've slept with a woman, everything that happens after, even the most nightmarish abuse and family terrorism, is viewed as something they brought on themselves. All they do is shut their mouth, get a lawyer, and say goodbye to half their assets and income.
professorgerm
You shall love your crooked neighbor, with your crooked heart
7hr ago
The second was later enough in the day there were actually other clients there, but I'm still absolutely clueless and after how badly the first time went I brought a book, so when a guy sat next to me and ordered me a drink I said nope
Uff da, been there with the book.
I'm glad that on a couple other "just how oblivious can I be" occasions I had thoughtful friends nearby that recognized how dire the situation and coached me before it was too late and the opportunity lost.
Aapje58
8hr ago
Toilet wine is not agriculture. It's a mere conversion of one food stuff to another, and doesn't produce nutrients. Toilet wine is made from food given to the prisoners by the guards, so it's a very poor argument in the context of food self-sufficiency of Gaza.
Keep in mind that Gaza is a desert region, so farming there is not easy. Especially since Gaza lies at the sea, so you have salt-water intrusion into the ground water. And the various disruptive behaviors of the Israeli settlers and government goes back for decades, which makes it a lot harder to farm. The area is also heavily overpopulated, in part due to the Israeli policy of taking ever more land from the Palestinians. The population density of Gaza is slightly smaller than of Hong Kong, so it is effectively a city state. It is not reasonable to expect much agriculture with that level of population density.
Gaza seems to have been a total economic basketcase going back decades.
Do you really think that it is reasonable to expect anything else given the conditions during those decades? For example, Israel never allowed Gaza to build a harbor so they could trade with other nations. If you were in charge in Gaza, how would you create a healthy economy?
Capital_Room
rather dementor-like
8hr ago
non-human persons
Is there such a thing? I mean, AIUI, unless you're talking about the legal construct that is the "legal personhood" of things like corporations (and I don't think you are), modern US law says humanity is a necessary (but not sufficient) condition for personhood; and, also AIUI, most countries aren't much different. (I've seen this discussed in the context of legal issues surrounding potential future contact with intelligent alien life, including the claim that the branch of the US Federal government with the proper legal authority over such contact would be the Fish and Wildlife Service.)
Patriarchy wouldn’t have applied to begin with; these aren’t chaste Virgin daughters he’s laying with. They would, by the standards of a patriarchal society, have been reckoned as prostitutes.
More options
Context Copy link