faul_sname
Fuck around once, find out once. Do it again, now it's science.
No bio...
User ID: 884
faul_sname
Fuck around once, find out once. Do it again, now it's science.
11mo ago
Are you referring to this giant 300 comment thread, or was there a more specific one?
faul_sname
Fuck around once, find out once. Do it again, now it's science.
11mo ago
Humans win because they are the most intelligent replicator. Winningness isn't an ontological property of humans. It is a property of being the most intelligent thing in the environment. Once that changes, the humans stop winning.
I mean I think humans win because they are the best at making and using tools (and they are the best at using tools partly because of their raw intelligence, but also partly because of other factors, including the runaway "better tools can be used to make better tools" process).
Of course, that's not super comforting since even modern not-that-finely-tuned language models are pretty good at making and using tools.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
11mo ago
See Gwern's terrorism is not effective. Thesis:
Terrorism is not about causing terror or casualties, but about other things. Evidence of this is the fact that, despite often considerable resources spent, most terrorists are incompetent, impulsive, prepare poorly for attacks, are inconsistent in planning, tend towards exotic & difficult forms of attack such as bombings, and in practice ineffective: the modal number of casualties per terrorist attack is near-zero, and global terrorist annual casualty have been a rounding error for decades. This is despite the fact that there are many examples of extremely destructive easily-performed potential acts of terrorism, such as poisoning food supplies or renting large trucks & running crowds over or engaging in sporadic sniper attacks.
He notes that a terrorist group using the obvious plan of "buy a sniper rifle and kill one random person per member of the terrorist group per month" would be orders of magnitude more effective at killing people than the track record of actual terrorists (where in fact 65% of terrorist attacks do not even injure a single other person), while also being much more, well, terrifying.
One possible explanation is given by Philip Bobbitt’s Terror and Consent – the propaganda of the deed is more effective when the killings are spectacular (even if inefficient). The dead bodies aren’t really the goal.
But is this really plausible? Try to consider the terrorist-sniper plan I suggest above. Imagine that 20 unknown & anonymous people are, every month, killing one person in a tri-state area28. There’s no reason, there’s no rationale. The killings happen like clockwork once a month. The government is powerless to do anything about it, but their national & local responses are tremendously expensive (as they are hiring security forces and buying equipment like mad). The killings can happen anywhere at any time; last month’s was at a Wal-mart in the neighboring town. The month before that, a kid coming out of the library. You haven’t even worked up the courage to read about the other 19 slayings last month by this group, and you know that as the month is ending next week another 20 are due. And you also know that this will go on indefinitely, and may even get worse—who’s to say this group isn’t recruiting and sending more snipers into the country?
Gwern concludes that dedicated, goal-driven terrorism basically never happens. I'm inclined to agree with him. We're fine because effectively nobody really wants to do as much damage as they can, not if it involves strategically and consistently doing something unrewarding and mildly inconvenient over a period of months to years (as would be required by the boring obvious route for bioterrorism).
I personally think the biggest risk of catastrophe comes from the risk that someone will accidentally do something disastrous (this is not limited to AI -- see gain-of-function research for a fun example).
faul_sname
Fuck around once, find out once. Do it again, now it's science.
11mo ago
Is your contention that more than one in a few tens of millions of people at most is strategically omnicidal ("strategically onmicidal" meaning "omnicidal and willing to make long-term plans and execute them consistently for years about it")?
I think the world would look quite different if there were a significant number of people strategically trying to do harm (as opposed to doing so on an impulse).
faul_sname
Fuck around once, find out once. Do it again, now it's science.
11mo ago
The following will get you, segmented by month,
-
Number of signups in that month
-
Number of those signups who have ever commented
-
Average number of comments by those signups (total and per day)
-
Average number of comment words written by those signups (total and per day)
-
Average number of upvotes on comments by those signups (total and per day)
-
Average number of downvotes on comments by those signups (total and per day)
-
Average score (up - down) on comments by those signups (total and per day)
with user_comment_stats as (
select
u.id as user_id,
min(to_timestamp(u.created_utc)::date) as signup_date,
count(c.id) as n_comments,
sum(array_length(regexp_split_to_array(trim(c.body), E'\\W+'), 1)) as total_words_written,
sum(c.upvotes) as total_upvotes,
sum(c.downvotes) as total_downvotes,
sum(c.upvotes - c.downvotes) as total_score,
min(now()::date - to_timestamp(u.created_utc)::date) as days_since_signup
from users u
left join comments c on c.author_id = u.id
group by user_id
)
select
left(signup_date::text, 7) as signup_month,
count(distinct user_id) as n_users,
sum((n_comments > 0)::int) as n_commented,
avg(n_comments) as avg_comments_per_user,
avg(total_words_written) as avg_commentwords_per_user,
avg(total_upvotes) as avg_upvotes_per_user,
avg(total_downvotes) as avg_downvotes_per_user,
avg(total_score) as avg_score_per_user,
avg(n_comments / days_since_signup) as avg_daily_comments_per_user,
avg(total_words_written / days_since_signup) as avg_daily_commentwords_per_user,
avg(total_upvotes / days_since_signup) as avg_daily_upvotes_per_user,
avg(total_downvotes / days_since_signup) as avg_daily_downvotes_per_user,
avg(total_score / days_since_signup) as avg_daily_score_per_user
from user_comment_stats
group by signup_month
You can do a similar thing with votes and posts if you care about such things.
Though honestly there are fewer than 2500 users and 100000 comments on the motte -- fancy analytics approaches are unlikely to tell you anything you don't already know from regularly reading the motte.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
11mo ago
Yudkowsky is worried about nothing! All we have to do to solve the alignment problem is make sure that the AI can use humans effectively as tools to accomplish its goals.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
11mo ago
I think it looks like someone complaining on the internet that something is BROKEN and BAD and they want it to be GOOD INSTEAD, and then having someone else fixes it through means they don't understand.
Anyway, on a completely unrelated note when I click "reply" on a comment that is at or past the depth limit (e.g. 8+ comments deep in the CW thread), and I click the button that says "comment", the button gets disabled but it never shows that it successfully posted (though it did successfully post). This is BROKEN and BAD and I want it to be GOOD INSTEAD.
Meow.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
11mo ago
On the flip side I think "GPT-as-a-linter-for-contracts" might be a killer application. Take a set of "things that are frequently bad in contracts", for each item in the list and each clause of the contract ask if that clause of the contract has that particular issue.
If a contract passes the linting step that doesn't necessarily mean that it's fine, but if it fails the linting step that's at the very least a sign that a human should be looking at that particular part more closesly.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
11mo ago
That programming language sounds like a nightmare to program in.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
11mo ago
I think "the general factor g of intelligence" is just "the principle component if you do principle component analysis on things that purport to measure intelligence". Like it definitely exists in the sense that PCA does indeed spit out a large first component if you do that. As long as all of the intelligence tests you're feeding into the statistical process meet some specific criteria (e.g. linearity, similar scale), it very strongly demonstrates that all of the tests are mostly measuring the same thing as each other.
To develop an intuition for what "the PCA spit out a large primary component" means in practice, let's consider an example housing dataset which includes attributes like price, number of beds / baths / parking spaces, presence of AC/heat, etc. If you do a PCA on that dataset, you get a primary component which explains 25% of the variance (and the next components explain 12%, 9%, 9%). Let's call this primary component the "general factor h of house-goodness". Sale price, number of bedrooms, and presence of air conditioning, in particular, are very strongly h-loaded, though every thing that would be nice to have in a house is positively h-loaded.
It's pretty clear that h reflects an actual thing, and that actual thing is probably approximated by "how good the house is". It's sensible talk about things like "high h houses that score low on the price test", or, as we say in plain language "nice houses that are cheap".
What the presence of h does not mean is anything like "this house scores higher on the number of bedrooms test because it has a high h factor".
You should view "the general factor g of intelligence" through that lens.
(cc @RococoBasilica, this seems relevant to your questions)
faul_sname
Fuck around once, find out once. Do it again, now it's science.
11mo ago
I'm convinced a sufficiently smart AI could build and deploy nanobots in the manner Yud proposes.
I'm not convinced that's possible. Specifically I suspect that if you build a nanobot that can self-replicate with high fidelity and store chemical energy internally, you will pretty quickly end up with biological life that can use the grey goo as food.
Biological life is already self-replicating nanotech, optimized by a billion years of gradient descent. An AI can almost certainly design something better for any particular niche, but probably not something that is simultaneously better in every niche.
Though note that "nanobots are not a viable route to exterminating humans" doesn't mean "exterminating humans is impossible". The good old "drop a sufficiently large rock on the earth" method would work
faul_sname
Fuck around once, find out once. Do it again, now it's science.
11mo ago
· Edited 11mo ago
Nanobots presumably would be even more flexible.
Why would we presume this? Self-replicating nanobots are operating under the constraint that they have to faithfully replicate themselves, so they need to contain all of the information required for their operation across all possible environments. Or at least they need to operate under that constraint if you want them to be useful nanobots. Biological life is under no such constraint. This is incidentally why industrial bioprocesses are so finicky: it's easy to insert a gene into an E. coli that makes it produce your substance of interest, but hard to ensure that none of the E. coli mutate to no longer produce your substance of interest, and promptly outcompete the ones doing useful work.
why not a virus that spreads to the entire population while laying dormant for years and then start killing, extremely light viruses that can spread airborne to the entire planet, plenty of creative ways to spread to everyone not even including the zombie virus
I don't think I count "machine that can replicate itself by using the machinery of the host" as "nanotech". I think that's just a normal virus. And yes, a sufficiently bad one of those could make human civilization no longer an active threat. "Spreads to the entire population while laying dormant for years [while not failing to infect some people due to immune system quirks or triggering early in some people]" is a much bigger ask than you think it is, but also you don't actually need that, observe that COVID was orders of magnitude off from the worst it could have been and despite that it was still a complete clusterfuck.
Although I think, in terms of effectiveness relative to difficulty, "sufficiently large rock or equivalent" is still wins over gray goo. Though there are also other obvious approaches like "take over twitter accounts of top leaders, trigger global war". Though probably it's really hard to just take over prominent twitter accounts.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
11mo ago
· Edited 11mo ago
I personally can't see how eyes might evolve for the first time but I accept that it happened.
Off topic, but this is my area so I can't resist.
The key thing to understand about the evolution of complex biological systems is that they didn't just pop up fully-formed. Instead, they evolved through a series of small changes to simpler systems. The changes which worked better than the original were passed on to the next generation, and the next generation had a more functional but more complex version of that system. The story of the eye's development goes something like this:
Billions of years ago, a cell contained some retinal. Retinal is a simple molecule that has a special property: when light hits it, one of its double bonds can switch between the cis and trans conformations. The cell can then extract energy (in the form of a proton gradient across a membrane) by flipping it back. Being able to harvest energy from light was a massive advantage, and this adaptation spread like wildfire. An explosion of thriving life turned the world purple.
Once cells had retinal, their internal chemistry would change depending on whether they were exposed to light. This allowed cells to adapt their behavior based on the presence or absence of light.
Fast forward to multicellular life. Organisms could save energy by having cells express only the proteins they needed for their specific functions. The PAX genes allowed for specialization of cells based on their location, and clusters of photosensitive cells evolved into the first eye spots. If an animal had two eye spots on different sides of its body, it could tell the direction of light, helping it orient which way was up.
Over time, eye spots became cup-shaped, allowing them to distinguish light from more directions. The deeper the cup, the better it could do this. Eventually, the cups closed over at the top, turning into pinhole cameras with images projected onto a layer of photoreceptors at the back. Organisms like nautiluses can see blurry images with eyes like these.
Finally, lenses made of clear proteins with different refractive indices came along. Selective pressure favored organisms that could see better, pushing for the development of lenses—specifically, lenses shaped to focus light more precisely onto the retina.
So that's how eyes evolved for the first time—through steady selective pressure stacking small adaptations, one after the other, all the way from basic photosynthesis to the human eye.
Reality doesn't need to sound plausible to be true, but it usually does end up making sense once you understand the driving mechanisms.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
11mo ago
I've had a lot of luck asking for search terms rather than papers for literature review.
I have a feeling that lit review is going to be a super useful place for chatgpt plugins
faul_sname
Fuck around once, find out once. Do it again, now it's science.
11mo ago
Other threats have come (nuclear weapons) and we've always come through them
I would actually really like to see a rebuttal of this one, because the doomer logic (which looks correct to me) implies that we should all have died decades ago in nuclear fire. Or, failing that, that we should all be dead of an engineered plague.
And yet here we are.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
11mo ago
I think the anthropic principle is fine for pointing out why we don't see things with bimodal outcomes of "everything is fine" / "everyone is dead".
But nuclear and biological weapons don't look like that. If 5% of worlds have no nuclear war, 40% have one that killed half the population, and the other 55% have one that wipes out everyone, 80% of observers should be in the "half of the population died in a nuclear war" worlds.
Which means one of the following:
-
Nuclear war will generally kill everyone in pretty short order (and thus by the anthropic principle most observers are in worlds where nuclear war has never started)
-
We're quite lucky even taking the anthropic principle into account: most observers are in more disastrous worlds than us
-
Nuclear war isn't actually very likely: most observers are in worlds where it never gets started
-
Something else weird is going on (e.g. simulation hypothesis is correct).
Hypothesis 1 seems unlikely to me since the models I've seen of even a full counter-value exchange don't seem to kill more than half the people in the world. Hypothesis 3 seems like the sort of world that does not contain the Cuban missile crisis, Petrov, the Norwegian rocket incident.
Which leaves us with the conclusion that either hypothesis 2 is correct and we're just lucky in a way that is not accounted for by the anthropic principle, or our world model has a giant gaping hole in it.
I think it's probably the "giant gaping hole" one. And so any doomer explanation that also would have predicted nuclear (or biological) doom has this hole. And it's that point I would like to see the doomers engage with.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
11mo ago
So I have two points of confusion here. The first point of confusion is that if I take game theory seriously, I conclude that we should have seen a one-sided nuclear war in the early 1950s that resulted in a monopolar world, or, failing that, a massive nuclear exchange later that left either 1 or 0 nuclear-capable sides at the end. The second point of confusion is that it looks to me like it should be pretty easy to perform enormously damaging actions with minimal effort, particularly through the use of biological weapons. These two points of confusion map pretty closely to the doomer talking points of instrumental convergence and the vulnerable world hypothesis.
For instrumental convergence, I will shamelessly steal a paragraph from wikipedia:
Agents can acquire resources by trade or by conquest. A rational agent will, by definition, choose whatever option will maximize its implicit utility function; therefore a rational agent will trade for a subset of another agent's resources only if outright seizing the resources is too risky or costly (compared with the gains from taking all the resources), or if some other element in its utility function bars it from the seizure. In the case of a powerful, self-interested, rational superintelligence interacting with a lesser intelligence, peaceful trade (rather than unilateral seizure) seems unnecessary and suboptimal, and therefore unlikely.
This sounds reasonable, right? Well, except now we apply it to nuclear weapons, and conclude that whichever nation first obtained nuclear weapons, if it wanted to obtain the best possible outcomes for itself and its people, would have to use their nuclear capabilities in order to establish and maintain dominance, and prevent anyone else from gaining nuclear capabilities. This is not a new take. John von Neumann was famously an advocate of a "preventive war" in which the US launched a massive preemptive strike against Russia in order to establish permanent control of the world and prevent a world which contained multiple nuclear powers. To quote:
With the Russians it is not a question of whether but of when. If you say why not bomb them tomorrow, I say why not today? If you say today at 5 o'clock, I say why not one o'clock?
And yet, 70 years later, there has been no preemptive nuclear strike. The world contains at least 9 countries that have built nuclear weapons, and a handful more that either have them or could have them in short order. And I think that this world, with its collection of not-particularly-aligned-with-each-other nuclear powers, is freer, more prosperous, and even more peaceful than the one that von Neumann envisioned.
In terms of the vulnerable world hypothesis, my point of confusion is that biological weapons actually look pretty easy to make without having to do anything fancy, as far as I can tell. And in fact there was a whole thing back in 2014 with some researchers passaging a particularly deadly strain of bird flu through ferrets. The world heard about this not because there was a tribunal about bioweapon development, but because the scientists published a paper describing their methodology in great detail.
The consensus I've seen on LW and the EA forum are that an AI that is not perfectly aligned will inevitably kill us in order to prevent us from disrupting its plans, and that even if that's not the case we will kill ourselves in short order if we don't build an aligned god which will take enough control to prevent that. The arguments for both propositions do seem to me to be sound -- if I go through each point of the argument, they all seem broadly correct. And yet. I observe that, by that set of arguments, we should already be dead several times over in nuclear and biological catastrophes, and I observe that I am in fact here.
Which leads me to conclude that either we are astonishingly lucky in a way that cannot be accounted for by the anthropic principle (see my other comment), or that the LW doomer worldview has some hole in it that I have so far failed to identify.
It's not a very satisfying anti-doom argument. But it is one that I haven't seen a good rebuttal to.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
11mo ago
· Edited 11mo ago
I could try to draw finer distinctions between the situations of post-WW2 USA and a hypothetical superintelligent AI, but really the more important point is that the people making the decisions regarding the nukes were human, and humans trip over the "some element in its utility function bars the action" and "self-interested" segments of that text. (And, under most conceptions, the 'rational agent' part, though you could rescue that with certain views of how to model a human's utility function.)
My point was more that humans have achieved an outcome better than the one that naive game theory says is the best outcome possible. If you observe a situation, and then come up with some math to model the situation, and then use that math to determine the provably optimal strategy for that situation, and then you look at what the actual outcomes of the situation and the actors obtain an outcome better than the one your model says is optimal, you should conclude that either the actors got very lucky or that your mathematical model does not properly model this situation.
And that's not even getting into how "having a fundamental psychology shaped by natural selection in an environment where not having any other humans around ever meant probable death and certain inability to reproduce their genes" changes your utility function in a way that alters what the game-theoretic optimal actions are.
I think you're correct that the "it would be bad if all other actors like me were dead" instinct is one of the central instincts which makes humans less inclined to use murder as a means to achieve their goals. I think another central instinct is "those who betray people who help them make bad allies, so I should certainly not pursue strategies that look like betrayal". But I don't think those instincts come from peculiarities of evolution as applied to savannah-dwelling apes. I think they are the result of evolution selecting for strategies that are generally effective in contexts where an actor has goals which can be better achieved with the help of other actors than by acting alone with no help.
And I think this captures the heart of my disagreement with Eliezer and friends -- they expect that the first AI to cross a certain threshold of intelligence will rapidly bootstrap itself to godlike intelligence without needing any external help to do so, and then with its godlike intelligence can avoid dealing with the supply chain problem that human civilization is built to solve. Since it can do that, it would have no reason to keep humans alive, and in fact keeping humans alive would represent a risk to it. As such, as soon as it established an ability to do stuff in the physical world, it would use that ability to kill any other actor that is capable of harming it (note that this is the parallel to von Neumann's "a nuclear power must prevent any other nuclear powers from arising, no matter the cost" take I referenced earlier).
And if the world does in fact look like one where the vast majority of the effort humanity puts into maintaining its supply chains is unnecessary, and actually a smart enough agent can just directly go from rocks to computer chips with self-replicating nanotech, and ALSO the world looks like one where there is some simple discoverable insight or set of insights which allows for training an AI with 3 or more orders of magnitude less compute, I think that threat model makes sense. But "self-replicating useful nanotech is easy" and "there is a massive algorithmic overhang and the curves are shaped such that the first agent to pass some of the overhang will pass all of it" are load bearing assumptions in that threat model. If either of them does not hold, we do not end up in a world where a single entity can unilaterally seize control of the future while maintaining the ability to do all the things it wants to.
TL;DR version: I observe that "attempt to unilaterally seize control of the world" has not been a winning strategy in the past, despite there being a point in the past when very smart people said it was the only possible winning path. I think that, despite the very smart people who are now asserting that it's the only possible winning path, it is still not the only possible winning path. There are worlds where it is a winning path because all paths are winning paths for that entity -- for example, worlds where a single entity is capable enough that there are no benefits for it of cooperating with others. I don't think we live in one of those worlds. In worlds where there isn't a single entity that overpowers everyone else, the game theory arguments still make sense, but also empirically doing the "not game-theoretically optimal" thing has given humanity better outcomes than doing the "game-theoretically optimal" thing, and I expect that a superintelligence would be able to do something that gave it outcomes that were at least that good.
BTW this comes down to the age-old FOOM debate. Millions of words have been written on this topic already (note that every word in that phrase is a different link to thousands-to-millions of words of debate on the topic). People who go into reading those agreeing with Yudkowsky tend to read those and think that Yudkowsky is obviously correct and his interlocutors are missing the point. People go into reading those disagreeing with Yudkowsky tend to read them and think that Yudkowsky is asserting that an unfalsifiable theory is true, and evading any questions that involve making concrete observations about what one would actually expect to see in the world. I expect that pattern would probably repeat here, so it's pretty unlikely that we'll come to a resolution that satisfies both of us here. Though I'm game to keep going for as long as you want to.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
11mo ago
I dunno, my highest voted comment ever had lots of formatting and upwards of 50 links.
And looking at my top comments in general, and also specifically those ones that have been AAQC'd, I see that "strong thesis supported by a bulleted or numbered list" seems to do well.
I do agree that eloquent and passionate rants also do well here, but I think that effortposts with backing links are also well-received when people put in the effort to make them.
Were you thinking of some particular well-formatted and linked post(s) that didn't get the support you expected it to recently?
faul_sname
Fuck around once, find out once. Do it again, now it's science.
11mo ago
Yeah, but the difference is that his boss was doing wrong and unethical things, and so doing whatever he said was a bad thing.
And honestly, on the meta level I don't even disagree with the sentiment that working to further the aims of a group can be good if the group is doing good things and bad if the group is doing bad things. It's not an incoherent position to hold. The people who are saying things like "protect trans children" do not see themselves as bad people who are bent on tearing apart the social fabric, that's how their opponents see them.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
11mo ago
I think you generally make a good point here.
Whether or not animals have qualia has no effect whatsoever on the causal progression of the universe.
This, though, I think is just factually wrong. The only reason "do animals have qualia" is a question we care about is because humans have qualia, and talk about those qualia, and thus the qualia of humans have an effect on the causal progression of the universe. If animals have qualia, it is as a part of their evolved responses to their environment, and it was only selected for to the extent that it causes their behavior to be better-suited for their environment.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
10mo ago
It is possible to seek asylum from SA to the US if you are prosecuted for homosexuality in SA.
Though if someone comes from SA seeking asylum, and are told they won't get it, and then marries a US citizen in the hopes of dodging extradition, I would at the very least question their judgment.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
10mo ago
Yeah, you can do it with circle of diameter 25mm, equilateral triangle with base of 25mm, square of side length 20 for example.
faul_sname
Fuck around once, find out once. Do it again, now it's science.
10mo ago
Surround it with backticks (`).
If you do it will say new
Edit: lol never mind, that's probably a bug.
Even triple backticks to make a code block don't work.
new
It's unspeakable.
There are a number of pro-bodily-autonomy-including-trans people on this site, myself included. There are a lot more people here who hold the position "body dysmorphia is bad" than "body dysmorphia is good", but that's because "body dysmorphia is good" is the straw "pro-trans" position [1]. I actually suspect that the following is a scissor statement here:
[1] Yes, I know that it is possible to find people who say something that approximates this. This is because, for any position, particularly about something political, it is possible to find at least one person who will support that position.
More options
Context Copy link