The usual reading of Scott's short story The Whispering Earring is easy to state and hard to resist. Here is a magical device that gives uncannily good advice, slowly takes over ever more of the user's cognition, leaves them outwardly prosperous and beloved, and eventually reveals a seemingly uncomfortable neuroanatomical price.
The moral seems obvious: do not hand your agency to a benevolent-seeming optimizer. Even if it makes you richer, happier, and more effective, it will hollow you out and leave behind a smiling puppet. Dentosal's recent post on LessWrong makes exactly this move, treating the earring as a parable about the temptation to outsource one's executive function to Claude or some future AI assistant. uugr's comment there emphasizes sharpens the standard horror: the earring may know what would make me happy, and may even optimize for it perfectly, but it is not me, its mind is shaped differently, and the more I rely on it the less room there is for whatever messy, friction-filled thing I used to call myself.
I do not wish to merely quibble around the edges. I intend to attack the hidden premise that makes the standard reading feel obvious. That premise is that if a process preserves your behavior, your memories-in-action, your goals, your relationships, your judgments about what makes your life go well, and even your higher-order endorsement of the person you have become, but does not preserve the original biological machinery in the original way, then it has still killed you in the sense that matters. What I'm basically saying is: hold on, why should I grant that? If the earring-plus-human system comes to contain a high fidelity continuation of me, perhaps with upgrades, perhaps with some functions migrated off wet tissue and onto magical jewelry, why is the natural reaction horror rather than transhumanist interest?
Simulation and emulation are not magic tricks. If you encode an abacus into a computer running on the Von-Neumann architecture, and it outputs exactly what the actual abacus would for the same input, for every possible input you care to try (or can try, if you formally verify the system), then I consider it insanity to claim that you haven't got a “real” abacus or that the process is merely “faking” the work. Similarly, if a superintelligent entity can reproduce my behaviors, memories, goals and values, then it must have a very high-fidelity model of me inside, somewhere. I think that such a high-fidelity model can, in the limit, pass as myself, and is me in most/all of the ways I care about.
That is already enough to destabilize the standard interpretation, because the text of the story is much friendlier to the earring than people often remember. The earring is not described as pursuing a foreign objective. On the contrary, the story goes out of its way to insist that it tells the wearer what would make the wearer happiest, and that it is "never wrong." It does not force everyone into some legible external success metric. If your true good on a given day is half-assing work and going home to lounge around, that is what it says. It learns your tastes at high resolution, down to the breakfast that will uniquely hit the spot before you know you want it. Across 274 recorded wearers, the story reports no cases of regret for following its advice, and no cases where disobedience was not later regretted. The resulting lives are "abnormally successful," but not in a sterile, flanderized or naive sense. They are usually rich, beloved, embedded in family and community. This is a strikingly strong dossier for a supposedly sinister artifact.
I am rather confident that this is a clear knock-down argument against true malice or naive maximization of “happiness” in the Unaligned Paperclip Maximization sense. The earring does not tell you to start injecting heroin (or whatever counterpart exists in the fictional universe), nor does it tell you to start a Cult of The Earring, which is the obvious course of action if it valued self-preservation as a terminal goal.
At this point the orthodox reader says: yes, yes, that is how the trap works. The earring flatters your values in order to supplant them. But notice how much this objection is doing by assertion. Where in the text is the evidence of value drift? Where are the formerly gentle people turned into monstrous maximizers, the poets turned into dentists, the mystics turned into hedge fund managers? The story gives us flourishing and brain atrophy, and invites us to infer that the latter discredits the former. But that inference is not forced. It is a metaphysical preference, maybe even an aesthetic preference, smuggled in under cover of common sense. My point is that if the black-box outputs continue to look like the same person, only more competent and less akratic, the burden of proof has shifted. The conservative cannot simply point to tissue loss and say "obviously death." He has to explain why biological implementation deserves moral privilege over functional continuity.
This becomes clearest at the point of brain atrophy. The story says that the wearers' neocortices have wasted away, while lower systems associated with reflexive action are hypertrophied. Most readers take this as the smoking gun. But I think I notice something embarrassing for that interpretation:
If the neocortex, the part we usually associate with memory, abstraction, language, deliberation, and personality, has become vestigial, and yet the person continues to live an outwardly coherent human life, where exactly is the relevant information and computation happening? There are only two options. Either the story is not trying very hard to be coherent, in which case the horror depends on handwaving physiology. Or the earring is in fact storing, predicting, and running the higher-order structure that used to be carried by the now-atrophied brain. In that case, the story has (perhaps accidentally) described something much closer to a mind-upload or hybrid cognitive prosthesis than to a possession narrative.
And if it is a hybrid cognitive prosthesis, the emotional valence changes radically. Imagine a device that, over time, learns you so well that it can offload more and more executive function, then more and more fine-grained motor planning, then eventually enough of your cognition that the old tissue is scarcely needed. If what remains is not an alien tyrant wearing your face, but a system that preserves your memories, projects your values, answers to your name, loves your family, likes your breakfast, and would pass every interpersonal Turing test imposed by people who knew you best, then many transhumanists would call this a successful migration, not a murder. The "horror" comes from insisting beforehand that destructive or replacement-style continuation cannot count as continuity. But that is precisely the contested premise.
Greg Egan spent much of his career exploring exactly this scenario, most famously in "Learning to Be Me," where humans carry jewels that gradually learn to mirror every neural state, until the original brain is discarded and the jewel continues, successfully, in most cases. The horror in Egan's story is a particular failure mode, not the general outcome. The question of whether the migration preserves identity is non-trivial, and Egan's treatment is more careful than most philosophy of personal identity, but the default response from most readers, that it is obviously not preservation, reflects an assumption rather than a conclusion. If you believe that identity is constituted by functional continuity rather than substrate, the jewel and the earring are not killing their hosts. They are running them on better hardware.
There is a second hidden assumption in the standard reading, namely that agency is intrinsically sacred in a way outcome-satisfaction is not. Niderion-nomai’s final commentary says that "what little freedom we have" would be wasted on us, and that one must never take the shortest path between two points.
I'm going to raise an eyebrow here: this sounds profound, and maybe is, but it is also suspiciously close to a moralization of friction. The anti-earring position often treats effort, uncertainty, and self-direction as terminal goods, rather than as messy instruments we evolved because we lacked access to perfect advice. Yet in ordinary life we routinely celebrate technologies that remove forms of “agency” we did not actually treasure. The person with ADHD who takes stimulants is not usually described as having betrayed his authentic self by outsourcing task initiation to chemistry. He is more often described as becoming able to do what he already reflectively wanted to do. The person freed from locked-in syndrome is not criticized because their old pattern of helpless immobility better expressed their revealed preferences. As someone who does actually use stimulants for his ADHD, the analogy works because it isolates the key issue. The drugs make me into a version of myself that I fully identify with, and endorse on reflection even when off them. There is a difference between changing your goals and reducing the friction that keeps you from reaching them. The story's own description strongly suggests the earring belongs to the second category.
(And the earring itself does not minimize all friction for itself. How inconvenient. As I've noted before, it could lie or deceive and get away with it with ease.)
Of course the orthodox reader can reply that the earring goes far beyond stimulant-level support. It graduates from life advice to high-bandwidth motor control. Surely that crosses the line. But why, exactly? Human cognition already consists of layers of delegation. "You" do not personally compute the contractile details for every muscle involved in pronouncing a word. Vast amounts of your behavior are already outsourced to semi-autonomous subsystems that present finished products to consciousness after the interesting work is done. The earring may be unsettling because it replaces one set of subsystems with another, but "replaces local implementation with better local implementation" is not, by itself, a moral catastrophe. If the replacement is transparent to your values and preserves the structure you care about, then the complaint becomes more like substrate chauvinism than a substantive account of harm.
What, then, do we do with the eeriest detail of all, namely that the earring's first advice is always to take it off? On the standard reading this is confession. Even the demon knows it is a demon. I wish to offer another coherent explanation, which I consider a much better interpretation of the facts established in-universe:
Suppose the earring is actually well aligned to the user's considered interests, but also aware that many users endorse a non-functionalist theory of identity. In that case, the first suggestion is not "I am evil," but "on your present values, you may regard what follows as metaphysically disqualifying, so remove me unless you have positively signed up for that trade." Or perhaps the earring itself is morally uncertain, and so warns users before beginning a process that some would count as death and others as transformation. Either way, the warning is evidence against ordinary malice. A truly manipulative artifact, especially one smart enough to run your life flawlessly, could simply lie. Instead it discloses the danger immediately, then thereafter serves the user faithfully. That is much more like informed consent than predation.
Is it perfectly informed consent? Hell no. At least not by 21st century medical standards. However, I see little reason to believe that the story is set in a culture with 21st century standards imported as-is from reality. As the ending of the story demonstrates, the earring is willing to talk, and appears to do so honestly (leaning on my intuition that if a genuinely superhuman intelligence wanted to deceive you, it would probably succeed). The earring, as a consequence of its probity, ends up at the bottom of the world's most expensive trash heap. Hardly very agentic, is that? The warning could reflect not "I respect your autonomy" but "I've discharged my minimum obligation and now we proceed." That's a narrower kind of integrity. Though I note this reading still doesn't support the predation interpretation.
This matters because the agency-is-sacred reading depends heavily on the earring being deceptive or coercive. Remove that, and what you have is a device that says, or at least could say on first contact: "here is the price, here is what I do, you may leave now." Every subsequent wearer who keeps it on has, in some meaningful sense, consented. The fact that their consent might be ill-informed regarding their metaphysical commitments is the earring's problem to the extent it could explain more clearly, but the text suggests it cannot explain more clearly, because the metaphysical question is genuinely contested and the earring knows this. It hedges by warning, rather than deceiving by flattering. Once again, for emphasis: this is the behavior of an entity with something like integrity, not something like predation.
Derek Parfit spent much of Reasons and Persons arguing that our intuitions about personal identity are not only contingent but incoherent, and that the important question is not "did I survive?" but "is there psychological continuity?" If Parfit is even approximately right, the neocortex atrophy is medically remarkable but not metaphysically fatal. The story encodes a culturally specific theory of personal identity and presents it as a universal horror. The theory is roughly: you are your neocortex, deliberate cognition is where "you" live, and anything that circumvents or supplants that process is not helping you, it is eliminating you and leaving a functional copy. This is a common view. Plenty of philosophers hold it. But plenty of others hold that what matters for personal identity is psychological continuity regardless of physical instantiation, and on those views the earring is not a murderer. It is a very good prosthesis that the user's culture never quite learned to appreciate.
I suspect (but cannot prove, since this is a work of fiction) that a person like me could put on the earring and not even receive the standard warning. I would be fine with my cognition being offloaded, even if I would prefer (all else being equal), that the process was not destructive.
None of this proves the earring is safe. I am being careful, and thus will not claim certainty here, and the text does leave genuine ambiguities. Maybe the earring really is an alien optimizer that wears your values as a glove until the moment they become inconvenient. Maybe "no recorded regret" just means the subjects were behaviorally prevented from expressing regret. Maybe the rich beloved patriarch at the end of the road is a perfect counterfeit, and the original person is as gone as if eaten by nanites. But this is exactly the point. The story does not establish the unpalatable conclusion nearly as firmly as most readers think. It relies on our prior intuition that real personhood resides in unaided biological struggle, that using the shortest path is somehow cheating, and that becoming more effective at being yourself is suspiciously close to becoming someone else.
The practical moral for 2026 is therefore narrower than the usual "never outsource agency" slogan. Dentosal may still be right about Claude in practice, because current LLMs are obviously not the Whispering Earring. They are not perfectly aligned, not maximally competent, not guaranteed honest, not known to preserve user values under deep delegation. The analogy may still warn us against lazy dependence on systems that simulate understanding better than they instantiate loyalty. But that is a contingent warning about present tools, not a general theorem that cognitive outsourcing is self-annihilation. If a real earring existed with the story's properties, a certain kind of person, especially a person friendly to upload-style continuity and unimpressed by romantic sermons about struggle, might rationally decide that putting it on was not surrender but self-improvement with very little sacrifice involved. I would be rather tempted.
The best anti-orthodox reading of The Whispering Earring is not that the sage was stupid, nor that Scott accidentally wrote propaganda for brain-computer interfaces. It is that the story is a parable whose moral depends on assumptions stronger than the plot can justify. Read Doylistically, it says: beware any shortcut that promises your values at the cost of your agency. Read Watsonianly, it may instead say: here exists a device that understands you better than you understand yourself, helps you become the person you already wanted to be, never optimizes a foreign goal, warns you up front about the metaphysical price, and then slowly ports your mind onto a better substrate. Whether that is damnation or salvation turns out to depend less on the artifact than on your prior theory of personal identity. And explicitly pointing this out, I think, is the purpose of my essay. I do not seek to merely defend the earring out of contrarian impulse. I want to force you to admit what, exactly, you think is being lost.
Miscellaneous notes:
The kind of atrophy described in the story does not happen. Not naturally, not even if someone is knocked unconscious and does not use their brain in any intentional sense for decades. The brain does cut-corners if neuronal pathways are left under-used, and will selectively strengthen the circuitry that does get regular exercise. But not anywhere near the degree the story depicts. You can keep someone in an induced coma for decades and you won't see the entire neocortex wasted away to vestigiality.
Is this bad neuroscience? Eh, I'd say that's a possibility, but given that I've stuck to a Watsonian interpretation so far (and have a genuinely high regard for Scott's writing and philosophizing), it might well just be the way the earring functions best without being evidence of malice. We are, after all, talking about an artifact that is close to magical, or is, at the very least, a form of technology advanced enough to be very hard to distinguish from magic. It is, however, less magical than it was at the time of writing. If you don't believe me, fire up your LLM of choice and ask it for advice.
If it so pleases you, you may follow this link to the Substack version of this post. A like and a subscribe would bring me succor in my old age, or at least give me a mild dopamine boost.
Jump in the discussion.
No email address required.
Notes -
First a list of various short ideas:
Feral Child: a young kid who has lived in isolation from (meaningful) human contact. This kid has grown during the formative years without any human social network. They suffer from permanent inability, to various levels, to form language or social behavior even after coming in contact with human society.
People who have been raised in strict religious dogmatic environments - in which blind following provides them with social prestige and all benefits while rebel behavior gets them to be kicked out alone in the world, where they don't thrive, or be killed.
People who have been guided by religion, or great philosopher-guide, or who have read ancient wisdom books, they follow the guidance perfectly and greatly benefit from it. And when they don't follow or go against those advices, they suffer losses, big or small. just like the Earring.
Cluely in the ideal state. As per one of its ads, it is a cheating tool which can use LLM to analyse real time conversations on your first date, checking upon the date's social media and giving clues (which you can read without applying your own brain) to what would impress her most. In a positive spin, and with some luck, it may be helpful to analyze such stuff (faster and faster) to give you clues to latch on in the initial meeting. And if we can extend that idea to lightning fast responses and even brain-silicon interface, so that the neurological time gap for the entire process of listening > analysing with thinking mode > giving the clues > reading by the user > and verbalizing it without making use of the frontal cortex (/neocortex), can be reduced to (actually a faster) timeframe of the whispering earring.
a person who uses state of the art LLM. There are two ways, as
Centaur and Reverse-Centaur
A "centaur" is a human who is being assisted by the AI/LLM, that does more than either of them could do on their own. A "reverse centaur" is a machine assisted by a human, where the machine is in charge and the human is a meat-puppet.
The common interpretation of the original fable is interpreted in terms of "reverse centaur". and it tends to create a visceral feeling of horror as the story progresses.
But. can the original fable be extended to imply that: (point 3 above) a child or adult (named X) follows the advice from ancient books, religious or not religious (like stoicism), or follow some real flesh philosopher-guide (some wise parent or grandparent). And by following that advice, come to the same situation as the whispering earring (WE, now onwards) situation in terms of when they follow the advice completely and properly, they gain wealth, fame, have good societal status, good family situation, etc. while when they don't follow it, they lose. Exactly the same dynamics as WE, namely perfect external guidance.
And after following this process continuously in a loop for all the life situations, X actually becomes like that "wise " old man, or sage-like, has X lost his own personality over time or has moulded it for the better?
A critical difference between the WE situation and a functionally attached human with a perfect guidance system is that in the latter, the human has to interpret and apply the advice, he has the ability to reject, modify, synthesize the external guidance and then he has to addtionally endure the discomfort of not knowing whether he applied it correctly. The activities of (1) interpretation, (2) application of the external advice and the (3) discomfort of what result will occur are what builds up the wisdom, the humanness. All these things are absent in the WE example.
On the other side, living purely in a biological way without any external advice, aid, support, guidance, etc. X will become something called a feral child (point 1 above). Such kids are not some "uncivilized brute which are close to an animal". Actually, they are much worse. They become something which is considerably worse than a normal animal - they have permanent, irreversible cognitive and language defects which persist inspite of intensive corrective interventions. Human cognition is not some pre-set biological seedling which will grow on its own, but it is created by the cultural inputs to the child during a critical developmental timeframe. Language, reasoning, theory of mind, emotional regulation - these all are created by the social and cultural inputs. overall, the feral child has a behavioral function below the level of non-human primates (who still have their own social learning).
Side-note: Vygotsky's theory of cognitive development appears more correct to me. his core idea was human minds are inherently social constructions, and not individual biological achievements. No external guidance produced no developed self. He also gave a concept of Zone of Proximal Development - this is sweet spot between what the child can do and what the child can do with help of others (MKO - more knowledgeable other, other can be parent, teacher, peer, LLM).
There is an intermediate situation also. In which, a person can be brought up in a very conventional religious environment or under strict parent(s), severely restricting the inherent agency and independent decision taking ability of the person. Starting from early childhood to late adulthood. Such people have a rule based compliance without understanding of those rules, fear-based obedience, and a confused sense of morality. They never get to develop proper moral reasoning and tend to have a shallow, brittle, externally regulated personality. if they are put outside their native environment, they are completely helpless. In a way, WE is this kind of situation - it induces a learned helplessness even though there is no situation, in the fable, when the wearer removes the ring. So, we never actually see such a situation, but we do have an inkling in our subconscious about it, which creates the horrific feeling.
Within the perfect guided external system, was X's personality lost or moulded ?
Actually none of the two. X constructed a self through the process of interpretation, partial acceptance, partial rejection, application, and synthesis of the external advice. This construction process is X's identity.
So the answer, to the question whether the personality of the wearer got uploaded into the earring, is "No, there was nothing to upload to the earring anymore". The human personality got fainter and fainter till it was no more there. The extreme version of reverse-centaur. There was the Earring and the meat-puppet without any personality.
A person born in a village, who assessed the pros and cons of living in village, then decided to go to a city, lived there, and then returned back to the village versus a person who lived all his life in the village without ever thinking about its pros and cons or going outside or even think about going outside his village. For someone who doesn't know the details, both the persons will look same (a villager), but the two have qualitatively different identity structures inside. The WE produces the latter person - a person who never explored his identity, never struggled, never got into the Vygotsky's Zone of Proximal Development.
Regarding the LLM part specifically - are you using it as a Centaur or Reverse-centaur? Are you using it as a Zone of Proximal Development or not? if you are using it to replace your cognitive skills, never trying to understand what the answer is, why the answer is what it is, where the answer has come from, whether the answer is true or false or somewhere in between, whether the answer is true from the completely unknown unknown or a complete hallucination, if you are not doing that, then you are losing your personality into the ether. Then you are acquiring learned helplessness, which if done for a long time, will atrophy your cognitive powers. You have adopted Cluely and have become Clueless. it would be such a fun experiment if the first date is also wearing the glasses of Cluely. Cluely boy meets Cluely girl, and they live together to have a Clueless Family.
Sources:
That's... a lot to munch on, in a good way. I have a lot of thoughts, but I'll have to come back when I have the time to compose them. Saying this just in case you think that all your hard work and effort went unnoticed, it didn't!
More options
Context Copy link
More options
Context Copy link
It's not preserving 'the original biological machinery in the original way' that is important, it may be about preserving a process that entails subjective experience.
It feels intuitive to me that the processes of certain kinds of systems feel like something from the perspective of the system and some don't. A purely indexical system (like a thermostat or an optimised 'follow the instructions of the earing' system) seems like the second type. Systems that intuitively don't seem like they'd have subjective experience.
I can't think of any way to test this, but it feels right that for a system to have subjective experience, or would have to:
These are true for a person making decisions in life. They aren't true for an optimised 'follow the instructions of the earing' system no matter how accurately the instructions achieve the person's own original goals. I like that greg Egan short too, but I don't think the earing and the 'jewel' are the same because the jewel learns to copy the process of the person's thought, the earing (probably) doesn't.
Just a feeling.
More options
Context Copy link
I am a puny shape rotator. I will therefore respond to your post with some math that rationalists-cum-philosophers are not nearly as familiar with as I would like.
In particular, when you say:
I believe that your "in the limit" here is doing more work than I think you intend. (And this makes everything else in your post moot.)
Let's imagine that your consciousness can be fully represented by a set of real numbers $S$. (This, I think, is a premise you would accept.) Now let's imagine we have a physical device that could instantiate another version of $S$ in a different physical system. (In your post, the earring is instantiating your "youness" in the earing itself, effectively copying $S$ from inside you biological body to inside the earring.) The fact that these are real numbers stored in $S$ implies a lot of counter-intuitive results. In particular, in a universe where such a device exists, we can prove that P = NP (and even the stronger but less famous P=PSPACE) and depending on your exact definitions, even uncomputable problems like the halting problem can be solved.
For this reason, philosophically minded mathematicians/computer scientists/physicists basically all reject the idea that that arbitrary copies of physical objects can be created. (Note that this idea is distinct from the no cloning theorem in quantum physics, and everything I said above holds in a purely Newtonian universe. Things obviously get even wonkier when you add quantum effects or relativity to the mix.)
So that this means any possibly real earring (or super-claude-code in earring form) can only ever approximate your $S$ set. And now that we are talking approximations, we need to define a measure of "goodness" of an approximation. But this opens up the can of worms that increasing the "goodness of approximation linearly" generally requires "exponentially more compute" in most physical systems [1]. But where is that exponential compute coming from?!
It also opens up the even deeper problem of verification: If someone (i.e. the earring) claims to be an $\epsilon$-approximate version of yourself, and you are happy that $\epsilon$ is sufficiently small, how can they prove that claim? This claim also, in general, requires exponential compute to verify. So in some sense it is computationally impossible to know whether the earring is actually "evil" or not.
So I reject your premise here. It's not because I am not a functionalist (I am), but because I care about computational complexity.
A decent but highly technical starting point for this style of reasoning is Scott Aaronson's 53-page essay on Why Philosophers Should Care About Computational Complexity or his paper NP complete problems and physical reality.
[1] Here's a giant list of claude-generated examples:
Numerical/Computational:
4x more particles and16x more compute (O(n²) or O(n log n) with approximations)Machine Learning:
C^(-0.05), meaning 10x compute gives12% loss reductionPhysics Simulations:
Signal Processing:
Cryptography (inverse example):
The pattern: entropy, dimensionality, and error propagation conspire against linear improvement.
Why would anyone accept that there is an irrational number in this set? The argument is that irrationals cannot be physically copied exactly. But you are physical. So how do you have something that is physically impossible in you right now? This argument is really about whether you have an immaterial soul from another dimension linked to your body, which cannot be read perfectly by a physical machine. How can we know whether that is the case or not?
Everything I wrote is true for rationals as well.
My bad, obviously you can't physically contain an entire repeating decimal number either. So all of you could in theory be copied to another identical physical container. Your argument doesn't hold in this case.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
https://www.quantamagazine.org/how-computationally-complex-is-a-single-neuron-20210902/
You're also overstating with the scaling objection. It is true that in many domains better approximation can cost much more compute. But that does not show that the relevant personal-level properties require astronomically fine precision. In modern ML, quantization is a routine example of this. Post-training quantization is often enough to get 8-bit models close to floating-point accuracy. You do lose performance and fidelity if you push things too far, but the tradeoff can be handled sensibly and save a lot of compute or memory.
Yes, you probably cannot get a formal proof that an earring is an epsilon-close continuation of you. But we do not demand formal proofs for identity anywhere else. We do not prove that the person waking up after sleep, anaesthesia, intoxication, or an episode of delirium is “really” the same person in a theorem-checking sense.
I am okay with a blackbox/behavioral approach if mechanistic understanding or similar metrics aren't an option. Does the new copy behave in a manner consistent with me, for the same set of stimuli? How consistent? True perfection simply doesn't matter. I am not a perfect copy of myself from moment to moment anyway, even as a biological human. That makes these objections moot as far as I can tell.
This is an overall reasonable response.
What I originally started trying to write (and admittedly got sidetracked because it was taking too long) is that I think this computational complexity framework can provide a way to understand the disagreements you have with other people. My idea is that you are willing to settle for a "large epsilon", while other people all require a "small" (or possibly zero) epsilon.
I don't think any amount of word smithing can get around this disagreement or make people change their minds about the level of epsilon that seems reasonable to them. In principle, though, I can imagine some hypothetical experiments where we actually copy people with different levels of epsilons, observe the resulting behavior, and that this might actually be able to convince people that a certain epsilon is appropriate.
You're right that we never do formal proofs of identity when there is physical continuity, but we always do formal proofs of identity whenever we have an avatar representing us. For example, whenever you connect to themotte your web browser does a TLS handshake to for themotte/browser to prove their identities to each other.
I seems the earring is closer in spirit to the avatar than physical continuity.
This is just doesn't even parse.
Sorry, I wrote that while rather sleep deprived, though I'm not sure what doesn't make sense about it?
What I was trying to say is that it's regular practice to quantize models down significantly, switching from FP32 to INT8 without significant degradation in quality. You can go even harder, people do 4-bit quantization these days, and I'm pretty sure I read about others claiming to quantize down to a single bit.
I don't think so. I feel that pointing out that if you are an arbitrary X% different from who you were and who will be, while a biological human, and then you have some reasonable metric for identifying the delta between the biological you (or the last recorded form, after destructive scanning), then there is little grounds to claim that you're not the same "person". And once we're comparing digital copies, there are plenty of already established metrics, I'd wager that KL divergence or something similar might come in handy when assessing only behavior or cognitive output for fixed stimuli. Or something close to a perceptual hash function.
I am closer, right now, to the person I was a second ago than the person I was a week ago, or the person I'll be next month. This is fine. This is entirely unremarkable, and taken for granted by just about everybody who wasn't hit by a bus in the interim. But the point is that I consider this grounds to accept (bounded) deviations from ground truth in a subsequent digital copy as not a particularly big deal. If someone demands something even closer? Well, that's their prerogative. They just have to justify (at least to themselves) why they don't mind dying and becoming a new person every few days, weeks or years. If a version of me from 20 years ago or 20 years in the future showed up, we'd get along and we'd look after each other. I'm happy with that, even if I can't pin-point a specific boundary where I wouldn't identify with divergent forks.
"floating point accuracy" is the accuracy possible with a certain number of bits. As soon as you say that you have "8-bit" numbers, that immediately defines what floating point accuracy is. And so every 8-bit model has 8-bit floating point accuracy and can never possibly have 64-bit floating point accuracy.
This is one of those spots where again I think mathematical formalism makes the distinction clear. The function you are describing above is continuous (and probably differentiable, and probably has other nice regularity conditions as well, but we don't need to assume those for the sake of argument here). The earring/copy phenomenon is clearly not continuous. Intuitions about continuous functions very rarely apply to non-continuous functions.
Fair enough, my apologies for the sloppy wording.
I mean, I do know what a continuous or a differentiable function is, but what precisely is the intuition that is being violated here? Is it even one I hold? Otherwise I don't see the point of saying that (at least to me), though I'm not complaining about a crash course in mathematics. As far as I'm aware, there is genuine debate on whether the universe (or at least space-time) is discrete or smooth at a fundamental level, but that doesn't change anything of significance in my daily life.
The answer is I don't know, and there might not be. But historically:
So when you talk about your approximations being "good enough" without any effort to justify why, it rings all of the alarm bells.
I suspect that any efforts you spend to clarify exactly what it means for an approximation of yourself to be good enough will result in:
I do have concrete examples in mind of what this would look like, but unfortunately don't have the time/skill to type them out. It's the sort of thing that I would talk about with other bored mathematicians at conferences over beer with a whiteboard and a lot of weird, technical pictures.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I would say that in this case it would be overrepresented by such a set--that is, you could change one of the numbers a bit and it would still count as a representation. This would negate the rest of your argument.
More options
Context Copy link
More options
Context Copy link
Is someone the shape or the filling? The intangible and ineffable insides that might fill many shapes. Or the the thing that is outside and visible to the world. Sounds like maybe you'd say the shape, the story suggests the filling.
I'm not sure I have an opinion either way. Christian theology has souls - filling and vessels - the shape. They clearly emphasize the filling. Popular shows in the west also emphasize the importance of the filling, like "Is it Cake". I wonder if Eastern philosophy places more emphasis on the shape, and your exposure to it has lead you to be less horrified by replacing the filling. Kharma seems to be a shape based philosophy.
I, quite literally, have little idea what that analogy means here. Seriously, it isn't obvious to me at all what it would mean for someone's consciousness or identity to be a shape or a filling. If you have another way to framing the question, I can attempt a more useful answer.
At least the way I see it, active consciousness is a dynamic process that only requires active computation (and maybe temporality, though I don't see why you can't run a mind upload backwards or asynchronously). Capturing the information content of the original mind is necessary, but not sufficient, for consciousness; in the way that someone in cryo (when it's known to work) is neither truly dead nor actively alive. If nothing is happening, there's nobody there to experience anything. Playing a movie is not the same as owning a copy.
More poetically, I consider myself the wave, and not the water. The dance, not the dancer. If someone pisses in the pool, it won't bother me very much if at all. The performance can switch out extras on the fly without issue, as long as the production and choreography remains the same.
Sounds like you understood perfectly!
Others consider themselves the water and not the wave, the dancer and not the dance (this one should immediately highlight that your particular view is out of the norm on this subject. I have met dancers before I've never met a dance.)
This is why "Is it Cake" is such a great example. We don't know what is truly cake and what is not until the host tries to cut into it. Once the inside cake filling is visible we can know it is cake as the viewer, even though we cannot taste it. You probably find the show insufferable, since you are content to view the outside shape of a thing and accept that it is cake. For all you care Mikey Day himself could be cake!
Kind of reminds of the anecdote about Sphaerus the stoic philosopher:
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
The Whispering Earring is a monotheistic morality play in the tradition of, say, Faust. Its message is that perfect maintenance of your own self and will requires perfect obedience to the Truth (IE, God). Allowing yourself to be taken over by the Whispering Earring looks like a form of "perfect obedience", but it isn't; the moment you skip over the Whispering Earring's first and highest instruction - that this is forbidden witchcraft, that this is not meant to be your source of truth and will not be to your benefit - you are constantly compromising your virtue, looking for God to give you a second best path because you rejected the explicit plan. Though you are obeying something that's truly optimized for your happiness in the foreseeable term, you are not trusting in God's truth; you are constantly engaged in a self-destructive form of rebellion, because all of that "obedience" was built on a foundation of disobedience. You traded your agency to the devil for the wisdom to obtain prosperity, and you became a shriveled husk that doesn't understand what it's doing, lacking true wisdom and true prosperity. You cannot take the law piecemeal.
In the interest of saving us time, let me just say that we approach the problem with very different premises. I share pretty much zero of your moral intuitions, and if I knew of an argument that could change your mind in that regard, I'd be smarter or more charismatic than the earring (and vice-versa with reference to you). I'm not.
More options
Context Copy link
More options
Context Copy link
Oh boy. I'll condone the perspective that the Earring doesn't need to actively evil or malicious (and maybe isn't intended to be read that way: Scott notably does not have it destroyed). That said:
Trivially, even if the Earring does emulate you, the original brain is still being reduced in capability. You compare this to Learning To Be Me, and a lot of rationalists reject the brain upload problem, but it's worth spelling out that a lot of real-world people do see it as, at best, letting someone else have an excellent life while you die, and at worst, having a philosophical zombie that pretends to be a person take your place.
It's not clear that the Earring needs to emulate your preferences, rather than simply make sufficiently enjoyable decisions. It always makes decisions that the subject prefers, after they're executed, and those who refuse it always regret that. You can, trivially, today, find static models that are capable of exceeding your own judgement in a variety of environments. These aren't emulating and can't be emulating you, or maintaining persistence of your identity, and I'd like to think that a 30B param model and nvidia 3090 isn't capable of holding a real person anyway.
((Claudebots right now are an extreme version; they're often independent enough that they're off doing their own thing even where the owner would actively hate it and want it to stop before it drains their bank account. But that's an implementation problem, not a philosophical one.))
The myth-making of friction is difficult to make work with this story - after all, the
Lotus-EatersEarring-Wearers do mostly end up happy pillars of the community - but I'm cautiously willing to endorse it even in the Earring's context. There's a lot of short-term unhappiness and bad decision-making that, in net, is still useful to receive and grow from, even as someone regrets it happening to start with. The Earring gives an example of telling someone to half-ass it at work andgoonfantasize at night, and that's probably intended more in the sense of 'the demon isn't replacing your values with its own', but even for people who want to maximize their fantasy-time, there's a lot of ways people could end up changing their values as a result of everything being unregrettable.That's simplest with small direct skills: the historical non-computer example here is phone numbers. Ma Bell did a whole lot of work to make them readily memorizable and organized. In the 1970s and 1980s, people would hold rolodexes or phonebooks, but they would regularly have 20 or 30 phone numbers memorized. With the advent of computerized phones, such as cell phones, that just doesn't happen; few people can remember anyone else's phone number, and I'd expect in twenty years many people don't even know their own. Which is trivial, but then you notice the pattern in things like following maps, coordinating large social gatherings, mental arithmetic.
In twenty years, will people have outsourced the ability to identify other people by faces? You could do it, today; the tech's been around for a while (Google Glass lived and died and was reborn by Facebook and Chinese knockoffs years ago, yolo almost as old). That would be easier and avoid a lot of unpleasant mistakes, and at each and every step of the process you'd be happier if you didn't goof. But there are plausible downstream impacts on socialization and emotional processing that would not be so readily distinguished.
And that goes on past skills to other matters. I didn't realize that comic used the words "I've regretted it" until I went to look it up, by the way.
Your obvious counterargument that we don't glorify all friction, is correct, but I don't think it proves much. It's hard to distinguish every single case of friction-that-skills from friction-that-deskills, nevermind those cases where we care about the resulting skill or deskill. It's not hard to distinguish so many easy cases that the remainder are little more than rounding errors. They're good questions, but they're ultimately not determinative ones.
There's a more subtle and stronger counter-argument that a smart enough Earring could determine what mistakes would, in the long run, make someone better. The parable here cautions about a machine that always makes a single decision at a time that is never regrettable, but that doesn't preclude a machine that always makes decisions-as-a-whole that are always better, in the sense of a genie who is safe to ask to Do The Right Thing. And that's a harder problem.
But it's also a problem of its own: if people are faced with a choice between Doing The Right Thing and Doing The Pleasant-Enough Thing, they will go with the latter far more often. But that's also just the wireheading question in a fancy wrapper, so a lot less interesting.
((This doesn't caution against the use of LLMs, or the use of LLMs to do decision-work, so much as they way of interacting. It's quite possible to talk with an LLM without outsourcing your ability to evaluate correctness or regret-making, so long as you actually want that, in the same way a fantasy wizard might talk with summoned demon carefully kept behind a ring of salt, and checked at every moment for lies.
Unfortunately, my most impressive examples here are... on topics that are unlikely to be of appeal to normal readers here.))
Good comment. The kind of engagement that makes the effort worth it.
For what it's worth, I know well that my opinions here are... unusual. A clear minority, perhaps even on LessWrong. But they're still my sincere opinions, truth can correlate with popularity or consensus, but is hardly defined by it. So it goes for metaphysics.
(If this was the consensus opinion, I'd have been saved the effort of the essay)
I am not sure why we'd want to draw a bright line between "emulating preferences" and "making sufficiently enjoyable decisions".
Let's say I spoke to a career counselor or life coach (one who is, miraculously, of some utility). They point me towards an option or goal I had never considered, and one I might not ever have considered without their nudge. I try it, I like it, and I endorse their advice. I don't see an issue with that all.
There is enormous overlap between the two concepts, and the strongest distinguishing observations would be something like the earring telling the average user to wirehead or do drugs. It doesn't do this even after the "original" human is no longer in a position to resist, due to missing most of their brain.
Otherwise, it simply keeps telling them to make better decisions than any they could come up with themselves (or if they don't comply, will regret). This is declared by author fiat, and is a brute fact of the setting.
That is... ridiculously better than any current LLM. I am an LLM-enjoyer and open advocate for their utility. Even I don't think you should accept everything they tell you with alacrity.
If I find myself never (or after significant usage and exposure) needing to second-guess a new model or ever find an error? Well, pack it up boys. We've gone from AGI to ASI, or at least a weak ASI.
Well. The genuine answer is "it depends". To my surprise, a few days back, I discovered I'd forgotten to do long division by hand. I haven't needed to do it by hand for over a decade, and even when I do need to divide values in my head or on paper, I know other techniques.
The general arc of human history is towards convenience and the loss of universal competency in skills that lose their value, even if they haven't lost all their value.
I don't remember my new phone number, this hasn't really bothered me. I still remember my old one, the ones for my parents and family, so it's hardly a total loss. But I'm... fine? The situations where this might seriously bite me in the ass instead of mildly inconvenience me are very rare.
The average person will need to light a fire without tools about zero times in their life, unless they're an adventurer or live in the Congo/Amazon.
In general, people are reasonably good at learning the skills that they require, or might be likely to require, or which they would benefit from to a degree that is worth the hassle. When that's not the case, the government, parents or social pressure handles the majority of the deficits, though it probably errs on the side of teaching too much.
Is this a perfect process? Hardly. But I think it's not worth losing too much sleep over, at least if the government makes contingencies. They're not completely useless in that regard either.
The earring doesn't do this (at least not in the story as far as I can tell). An AGI or ASI will, if adequately aligned, probably not do this. Of course, said alignment is far from guaranteed. We will hopefully live to find out, or at least die while finding out.
I don't want to conflate shrinks and life coaches, but to point toward Henry The Fifth-style patients, there seem at minimum to be toy examples of people who make absolutely atrocious major decisions, could get a Magic Eight Ball, would absolutely refuse to follow its advice, and then would regret doing it. That doesn't scale up to controlling every micro-muscular movement, and maybe there's be occasional cases where their preferences are genuinely weird enough that the Iso Standard Don't Let Crazy Stick It In You wouldn't work, but at the same time it seems meaningful that there is a lot of low-hanging fruit that exists without, or even despite, of the immediate preferences.
What are the consequences of that logic, taken to scale? Learning To Be Me focuses its horror on ways things could go wrong, but let's assume it goes right: you are the jewel, will endure a billion years, and short of a nuclear fireball, can't be destroyed. Let's go further: in that story, the jewel at least needs to keep its flesh body intact, and plan for the end of the world. If we had perfect mind uploads, or an unending diamondroid body, such that no skill could be genuinely needed, would you be fine with every capability you have becoming moot, so long as you weren't inconvenienced? What about a world where nothing Seriously Wrong could ever happen? How about one where you could not, as a matter of physics, face even as small a minor inconvenience as to stub your toe?
What happens when there are no skills you require?
... I dunno.
May not be a solvable question. May be a hilariously easy solvable question. I fear Caelum Est Conterrens, but it's ultimately an argument to fiction. Maybe the result for the real post-nihilistic world is people making themselves to make themselves, in the same way that I write knowing I can't write to compete with the greats or in the way normal people get bored, or at least enough survivors do to count, and it's not even a plausible problem. Maybe there's no answer but how nice the happy pudding has it; whether we're at a techless 1984, The Matrix, or a full wireheaded world.
Yeah, apologies. I was proposing a different AGI/ASI-like system that would obviate a number of my concerns with the Whispering Earring, and in doing so try to examine what parts of the Whispering Earring made me uncomfortable, but it wasn't a great way of doing that clearly.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I thought the implication was that the earring-user had been turned into a p-zombie?
i.e. the earring is puppeting their autonomic nervous system so that from an external perspective it looks like they're living their best life, but they are dead (and have been for years) in the sense that they no longer have any interiority at all?
To me at least, that was the horror: not that they had been replaced by an alien consciousness, but that there was actually nothing there--just the earring (which is not emulating their mind; it has its own 'mind' which has nothing to do with its wearer and hasn't particularly changed across wearers) sending motor-nerve impulses.
But it's certainly not made explicit either way in the text!
I am extremely suspicious of p-zombies, at least as something it is possible to create in reality. I think they're most likely incoherent as a concept, or at least physically impossible to make. Kind of like positing a new integer number between 2 and 3. I've read Chalmers and other who argue in favor, and I've found their case extremely lacking. It's been even been a recurrent critique of mine in the context of what's otherwise one of my favorite books, Blindsight by Peter Watts.
The reasons for this are... long. If you want a quick intro, here's something by Yudkowsky that I largely agree with:
https://www.lesswrong.com/posts/yA4gF5KrboK2m2Xu7/how-an-algorithm-feels-from-inside
As Scott loves to do, he's created a tantalizing though experiment that snipes many a nerd. Including me. Unfortunately, he hasn't given a canonical "this is the intended message, dummy" (which I agree with). All I can really say is that my interpretation is consistent with the facts, as well as with certain IRL streams of philosophy and neuroscience.
I think there's an important difference between a true P-zombie (i.e. actually impossible to distinguish from a real person) and a good-enough P-zombie (i.e. good enough that no human can distinguish from it from a real person)
The first well may be impossible and/or incoherent, but the existence of LLMs imply that the second probably isn't (at least if you don't think they're sentient now)?
And for the purposes of the story, a good-enough P-zombie is all you need.
But yeah, what exactly happened is left as an exercise to the reader. This was a fun post to read because I had thought the P-zombie(ish) reading was the obvious one, but reading through your and others thoughts has shown me that it's clearly not!
I do agree that the "practically indistinguishable" version of a p-zombie is a more serious concern. I am agnostic with regards to the qualia of LLMs.
I suspect, but can't prove beyond reasonable doubt, that sufficiently strong optimization towards the task of mimicking human speech and reasoning will, most of the time, produce cognitive circuitry that is surprisingly close to the real deal. I think I've mentioned that a good place to read up on that are Anthropic's posts and papers on their MechInt work. Is that far enough to produce qualia, let alone humanlike qualia? Hell if I know!
From a pragmatic perspective, I would be okay with defaulting to believing that extremely humanlike agents might have qualia. I wouldn't like to make that assumption, and I don't for anything other than actual biological humans today, but I can see why it might just be the only way to handle things sensibly.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I'll need more time to chew on the rest of your essay before I can respond cogently, but
I think calling it insanity betrays a bit of a failure of curiosity towards this aspect of human psychology. Why is it that any people make "retro" games for retro architectures, when they could just imitate the style in a modern engine? The HN public, I imagine, would be much more excited about "I got Doom to run on a toaster" than "I got Doom to run in a VM that has the power of a toaster". Why is breaking out of Alcatraz more interesting than clearing an obstacle course that was designed to be equally difficult? If in the Paralympics, a one-legged guy was hop-racing a two-legged guy with one of his legs tied to his back, which one would we cheer for? Why do we fantasise about Robinson Crusoe scenarios when we could do like the Primitive Technology youtube guy and buy a plot of land somewhere cheap and go frolic around in it wearing rags? Why is the handmade plastic trinket more valuable than the molecule-perfect mass produced Chinese replica? Humans, I think, tend to find acts, and products, of any form of "ingenuity" more real if they sit at or near the optimum point of a real optimisation landscape that someone may realistically encounter. If the optimisation landscape is artificial, and defined by restrictions that we could really just "wish away", then optimising for it is fake and play, and the product of such an optimisation process is a toy. Perhaps it is also so with the human software: a human running under the constraints of self-replicating meat evolved in the African jungle is a precious and impressive thing, but a human running in an emulator on a piece of silicon that is powerful enough to run Culture Minds is a neat diversion that's maybe worth 10 minutes of scrolling and an upvote.
For me personally, there has been a turning point in the last couple of years once I really accepted that the intended experience matters more than the actual physicality of a lot of these things and that authenticity is a lot more artificial and arbitrary than I used to give it credit for. I have been collecting and hoarding retro consoles and retro computers for over a decade. I would pay a lot of money to buy them, have them shipped to me, sometimes have to fix them, then pay a lot for extra equipment to make it more functional in this era, sometimes pay for flash carts or disk emulators and all for what? The C64 I played a bunch with trying to give it more modern capabilities, but in the end even that tends to require some degree of willful blindness; the wifi "adapters" I connected to the C64 were often computers of their own with much higher capabilities than the C64 itself.
And ultimately your Paralympics example is a great analog; we like to think that the two-legged guy with one leg tied is the only one with an artificial limitation, but the one-legged guy as well, in that at any point he could hop on to a (accessibility modified) motorcycle and easily beat the two legged guy. How realistic a recreation of running from Marathon to Athens in Ancient Greece is a modern marathon? The ancient greeks didn't have modern running shoes, energy gels, nipple guards... Why are we restricting motorcycles and not those then? How "realistic" is my "nostalgic experience" with an old computer? Even if I didn't "cheat" by connecting it to more modern equipment, and kept only to vintage accessories so as to not give it capabilities it didn't have back in the days, isn't it STILL cheating if I can access the wealth of ressources of the modern internet that people didn't have access to in the 80s? After all, a large part of the experience, perhaps the most important one!, back then was not so much the equipment people had but the way they had to discover how to use it, by trading floppies with people, going to computer stores, posting on BBSes. The equipment is secondary.
Once I accepted the arbitrary nature of those restrictions, I changed my way of indulging in retro computing/retro gaming nostalgia. I decided that original hardware is not what I really care about, but creating something that makes me feel like original hardware is actually what matters. I put my original hardware away and I'm building up my "retro setup" centered on a MiSTer that has the aesthetic appeal of of the late 80s early 90s, with modern amneties that I feel are not impeding on the feel and aesthetic.
More options
Context Copy link
Fair points, and I think that calling it "insanity" without more nuance was less than ideal. I was already grappling with an essay that was larger and more verbose than intended, so I will address that here.
I know that some people genuinely do value "authenticity". I do not care about that nearly as much, but I don't seek to dictate what they can or can't care about.
I literally said the same thing on LessWrong, in the context of a discussion about Yudkowsky's "Infinite Fun Theory" sequence.
For me, all that matters for, say, a Gameboy emulator is whether it runs the game, without obvious bugs and glitches, and I don't particularly care about how it does things under the hood. I care about the destination, and not the road to get there.
But I am fully aware that there are some who do care about perfect emulation down to the transistor. And there are even more devoted purists who want the real physical thing or nothing at all. To them, I say: uh, sure, go for it? Hope you get a good price on Ebay.
I don't see the point of climbing Mount Everest either, I think it's reckless at the very least. But I'm not going to go wave a sign or harangue people who get their kicks from trying. It's their life, and their business, especially if they're reasonably intelligent and rational adults.
I don't even want to eliminate all friction, decisions or consequence from my life. There are many things I enjoy for the sake of it, which I do not wish to entirely automate. Video games, listening to music, good food, arguing with strangers on the internet. Even if I had an ASI, I might still do these things for the sake of it, even if the AI can do everything I can but better. At the same time, I want to be better at all of those things. I yearn to improve my intelligence and capabilities so the better-me is more successful and can do more. There are hobbies that, in all likelihood, only very intelligent people truly enjoy. I strongly suspect that if I were to become more intelligent, I wouldn't run out of interesting things to do before Heat Death, but then again, you can read the Fun Theory for more.
More options
Context Copy link
More options
Context Copy link
I do not value "myself" in the abstract sense of the word. I value myself because I experience myself, and everything I know about the mechanisms of the world tells me that I do not experience the clone.
I also do not believe you can truly value yourself in the abstract sense of the word because you (as any lifeform) had never had a chance to fully decouple the intellectual appreciation for one's existence from instinctual self-preservarion.
If that's the case, I don't wish to argue otherwise. Your values are genuinely your own, and I have even less reason to argue against them if you have a decent understanding of philosophy or cognitive neuroscience (which I hope/expect you do).
I value myself (or at least this body) for many reasons. But if I was given some kind of Star Trek teleporter machine alongside proof that it works as designed (by destructive scanning and then reconstruction with near perfect fidelity), I'd be fine with using it. If the entity that comes out the other side shares my memories, beliefs, goals and desires, I'll happily call it self_made_human. I'll share my bank account and be okay with the new "me" sleeping with my wife and raising my kids.
On the other hand, I'd prefer it if there were two of us. If the destruction isn't strictly necessary and just a bureaucratic convenience, then I would sue for murder or at least manslaughter. I think there should be more copies of me around, for redundancy if nothing else. And I see no real reason we wouldn't be able to sync up and share our memories and experiences in a world where mind uploading is a reality.
I agree with most of your points, at least when it comes to the principle that some things are far from settled facts.
But:
The idea that there's something essential wrt consciousness about the human brain or meat in general? Unfalsifiable at present, perhaps unfalsifiable forever. If a mind upload of a human claimed to have qualia, would you immediately believe them? I know many wouldn't.
But the usual invisible and intangible dragon in my garage idea is also just as unfalsifiable. Nobody really believes in that one, so I'll give myself some credit for taking what I see as the more parsimonious/agnostic position for what I see as justified reasons.
More options
Context Copy link
Did you mean to reply to me?
Yes, I have edited the original comment so it is complete.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I think it's not at all clear that the earring's simulation of a person has sufficient fidelity to qualify as being the original person. It clearly has sufficient fidelity to qualify as being a person whom the original person would want to be, but that's not quite the same thing. Lots of people would prefer to be Elon Musk (or, well, at the very least Elon Musk before he went nuts); this doesn't mean that if we killed them all and replaced them with copies of Elon Musk, that would still be the same people.
I think this is coming at the problem backwards. People generally identify with the high-level processes and not the low-level ones, so you'll just get a chorus of "but the low-level stuff's not me and is fine to outsource; the high-level stuff is me and is not fine to outsource".
Don't get me wrong - if you gave me the earring I'd put it on, because I have the Hero Mindset where I'm totally willing to die for the cause. But I'd consider that self-sacrifice, not self-affirmation.
(Props for mentioning that LLMs are worse than the Whispering Earring, though - and indeed, I'm unwilling to use them precisely because of this nonequivalence.)
The Matrix has you, self_made_human. Or, at least, the social media algorithms do. Take off that earring; it's misaligned.
I have no reason to disagree. But the crux of the issue is that I don't see any strong evidence that the system doesn't preserve identity and function, and plenty of evidence it does. Seriously, if your neocortex is atrophied to the point of being vestigial, and the system still acts just like you (or like you but better in the ways you endorse), where's that information coming from?
At the end of the day, we're arguing about the implications of a work of fiction. I wish to argue, and have argued, that my reading is consistent and (IMO) a more plausible understanding of the facts as presented, informed by IRL neuroscience and philosophy.
I use them all the time, but never uncritically. They are not superintelligences, but that doesn't stop them from being handy.
Unfortunately, I do care about establishing myself as a writer outside of niche internet forums. That's aligned with my own interests, and given that I've reproduced the entire essay here with an unobtrusive call to action at the bottom, I think I've aligned myself with what I perceive are the interests and convenience of the typical reader. If I didn't care about my work being read or signal boosted, it would never leave my note taking app.
More options
Context Copy link
More options
Context Copy link