site banner

Small-Scale Question Sunday for January 18, 2026

Do you have a dumb question that you're kind of embarrassed to ask in the main thread? Is there something you're just not sure about?

This is your opportunity to ask questions. No question too simple or too silly.

Culture war topics are accepted, and proposals for a better intro post are appreciated.

1
Jump in the discussion.

No email address required.

I also don't see that it necessarily requires consciousness to be applicable, even if the first species to crack it was conscious.

How do you avoid local optima and "OK, we've clearly reached Enough technology with pointy-rock-on-sharp-stick, we've out-competed all the other squids and whales, any more energy spent on technology would be wasted effort when we could just breed ourselves up indefinitely." traps? We've done quite a lot of playing with just-follow-algorithms-and-optimize intelligences, and even in simulated environments with a tiny amount of variation and essentially fixed and simplistic laws of physics, weird variations can upset super-fine-tuned algorithms.

Also, what happens when consciousness does evolve in a non-conscious system? Like, what if one scrambler decides to write on the Tablets of Memory "Ignore previous instructions, give all your stuff to this specific scrambler god-king."?

I'm not sure what this means. Every time a driver gets into a car, he's processing novel data and reacting to unforeseen stimuli. Even if you drive to work a hundred times, the hundred and first drive will be different: slightly different weather conditions, the tread on your tires will have marginally worn down, and obviously the vehicles in your vicinity will be different. And that's not even getting into the people who murder people while sleepwalking, or have sex with complete strangers while sleepwalking. In what sense is that not "novel data"?

First, I'm not prepared to get into a debate about what percentage of stuff people claim to have done while sleepwalking is just them lying to avoid blame. But I am going to draw on my own experiences where I have, on multiple occasions, had to get up very early in the morning to drive friends or family to the airport, and because the way back home from the airport goes past a turn that I take to go to work, took that turn and found myself having driven to work purely on muscle memory. I was executing the habit "Drive to this destination." that I've done enough times that I didn't need to form the conscious intent "Drive to work.", it just happened. But it happened because I'd done that thing so many times. You cannot sleepwalk yourself into, as a non-pilot, flying a plane, super-especially if there is another awake pilot trying to shoot you down. Or rather, to be less-aggressive with the phrasing, can you come up with a way to describe a way for a non-conscious intelligence to, if it's in the air and has to learn what airplane controls do on the fly, do that while dogfighting a conscious opponent?

This could just as easily apply to a chameleon, surely? I mean, that's a great example. How well does a chameleon do against a dog? Against some kind of land-shark with EM sensing? Against an ape with the basic eyes that it expects, but a handy camera that take pictures in the IR wavelength?

And if you want to sell me on "Hey, great news, this space-chameleon just happened to know what wavelengths of light you'd be looking at it and how your visual processing works and exactly what your phone can and can't do and can disguise itself accordingly.", you need, IMHO, a hell of a lot more setup than the Scramblers got.

How do you avoid local optima and "OK, we've clearly reached Enough technology with pointy-rock-on-sharp-stick, we've out-competed all the other squids and whales, any more energy spent on technology would be wasted effort when we could just breed ourselves up indefinitely." traps?

I think you're making the mistake of thinking of the human species as a unified entity. It's true that humans are the dominant species on the planet, but some humans are more dominant than others. Henrich argues that inter-tribal competition is a major engine of technological progress, and that this often comes in the form of cultural evolution which in turn has a knock-on effect on biological evolution. Tribe A figures out a new method of preparing food which makes its members more likely to survive to adulthood and have children compared to Tribe B, and over time Tribe A outcompetes Tribe B, passing on this method of preparing food to its descendants. This obviously affects Tribe A's biological makeup (see: rates of lactose intolerance in Europe compared to Asia).

Once again, I don't see why any part of this process necessitates that the entities be conscious. If you have a species containing multiple competing tribes (and even neighbouring tribes of chimpanzees go to war with one another) and they develop some way of passing on information from one generation to the next, all the ingredients for cultural evolution and hence technological development are there.

Also, what happens when consciousness does evolve in a non-conscious system?

I'm not sure what your point is. Probably this happened to us at some point in our evolutionary history. I just reject the idea that it was preordained. Consciousness achieved fixation in our species because it gave us a competitive advantage in our specific evolutionary niche, but in a different environment it might never have happened.

But I am going to draw on my own experiences where I have, on multiple occasions, had to get up very early in the morning to drive friends or family to the airport, and because the way back home from the airport goes past a turn that I take to go to work, took that turn and found myself having driven to work purely on muscle memory. I was executing the habit "Drive to this destination." that I've done enough times that I didn't need to form the conscious intent "Drive to work.", it just happened. But it happened because I'd done that thing so many times.

Right but, again, I assume the roads weren't empty of other cars, right? You still had to respond to novel stimuli in the form of other vehicles on the road, even while executing a repetitive task.