Be advised: this thread is not for serious in-depth discussion of weighty topics (we have a link for that), this thread is not for anything Culture War related. This thread is for Fun. You got jokes? Share 'em. You got silly questions? Ask 'em.
- 113
- 2
What is this place?
This website is a place for people who want to move past shady thinking and test their ideas in a
court of people who don't all share the same biases. Our goal is to
optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.
The weekly Culture War threads host the most
controversial topics and are the most visible aspect of The Motte. However, many other topics are
appropriate here. We encourage people to post anything related to science, politics, or philosophy;
if in doubt, post!
Check out The Vault for an archive of old quality posts.
You are encouraged to crosspost these elsewhere.
Why are you called The Motte?
A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently,
it's an element in a rhetorical move called a "Motte-and-Bailey",
originally identified by
philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial
but high value claim to a defensible but less exciting one upon any resistance to the former. He likens
this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for
the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired
propositions to which one retreats when hard pressed."
On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.
New post guidelines
If you're posting something that isn't related to the culture war, we encourage you to post a thread for it.
A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts
such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a
submission statement. A submission statement is required for non-text sources (videos, podcasts, images).
Culture war posts go in the culture war thread; all links must either include a submission statement or
significant commentary. Bare links without those will be removed.
If in doubt, please post it!
Rules
- Courtesy
- Content
- Engagement
- When disagreeing with someone, state your objections explicitly.
- Proactively provide evidence in proportion to how partisan and inflammatory your claim might be.
- Accept temporary bans as a time-out, and don't attempt to rejoin the conversation until it's lifted.
- Don't attempt to build consensus or enforce ideological conformity.
- Write like everyone is reading and you want them to be included in the discussion.
- The Wildcard Rule
- The Metarule
Jump in the discussion.
No email address required.
Notes -
I believe in it, I just don't believe it will be that fast, especially at the "median human" skill level that I posited for the weaker of the two AGI definitions.
I'm having a somewhat hard time serializing my thoughts on this into a well-written argument, having drafted and re-drafted the rest of the post about three times now, so I'm sorry but I'll just give you some badly connected bullet points.
It's curious how the cutting edge of AI wound up being driven by computationally extremely costly black boxes that imitate human behaviour. Because of that, a lot of the standard intuitions about how AI will have a much easier time self-improving than humans because it will be a neat algorithm running on silicon are actually invalidated - a GPT-like AGI will operate as an inscrutable binary blob that inputs and outputs at bounded speed in an imprecise medium (text in human language), and resists parallelisation and other attempts at speeding up a single instance due to a myriad of superlinear factors and hardware barriers that are stumping teams of the world's top physicists, engineers and probably lab technicians holding illegible metis about how many times you have to bang the ASML machine with a wrench to get error-free DRAM, depending on the day's weather. I'm not convinced that generating more mediocre-human-tier ideas to try out is the primary bottleneck in this process, as opposed to something like "try changing the number of times you bang the machine, and if you break it, wait 6 months for ASML to build and deliver a new one" or "get the capital and paperwork lined up to even be allowed to try building a better chip".
There are billions of approximately average people, who are getting nowhere near innovating on cutting-edge AI or chip design. The weak AGI will probably require some pretty fancy high-end hardware to run; I don't think a billion copies of it will be available that soon after the 100 or 1000 or so it's first deployed on are. Due to the aforementioned curious nature of our current best candidates, N instances of them will probably scale a lot more similarly to "N humans" than to "an Nx as smart human". Changing the scaling will require solving a problem that so far has not proven easy for N mediocre humans to solve; drastically increasing the N will require physical-world stuff that the AGI can't do and therefore will continue advancing at meat human speed.
With current hardware, training times for new models are counted in weeks or months. Even if the million-mediocre AGI cluster generates a valid idea to slightly improve itself, it will take this long before it can start taking the million-slightly-less-mediocre AGI cluster online. This comes out of the same finite pool of resources: if it uses its numbers to perform "grad student descent" and generate 10 ideas of which one works without understanding one, this will take 10 times as long. We have no evidence that anything near our level can do better than grad student descent (i.e. identify some number of strategies/hyperparameters/? of which one randomly amounts to an improvement), and grad student descent is done with grad students who are much better than mediocre.
Nothing I've seen even from people near the top 10^-n fraction of the current human intelligence distribution has suggested to me that returns to intelligence are that insanely high. Von Neumann (even 1000 of him) could probably not have snapped his fingers and built GPT-4 in a year. Otherwise I think we would be seeing more divergence between countries right now than we do. Therefore I think that even as we approach the strong end of the weak-strong scale, acceleration won't be that great.
The sort of fiction that is enjoyed and written by our community betrays our biases on this matter: we always fantasize that if only we were 10% more intelligent, had access to that perfect motivator pill, had perfect memory or a magic notetaking and spaced repetition system, or some consciousness-forking powers, then the problems that we face day to day would just evaporate and turn the real world into a joyful, snappy, hairless RTS experience. (The positive framing of this, I believe, is called "internal locus of control".) A lot of strong AGI prognosis, in my eyes, winds up being coloured by some sort of bitter projection of what remains of the same fantasy: yes, one begrudgingly concedes, I will never become like Rationalist Harry and it was immature of me to think so - but the AGI will be just like Rationalist Harry, and it will leave you wishing you had gotten me as Rationalist Harry instead.
More options
Context Copy link