@faul_sname's banner p

faul_sname

Fuck around once, find out once. Do it again, now it's science.

1 follower   follows 1 user  
joined 2022 September 06 20:44:12 UTC

				

User ID: 884

faul_sname

Fuck around once, find out once. Do it again, now it's science.

1 follower   follows 1 user   joined 2022 September 06 20:44:12 UTC

					

No bio...


					

User ID: 884

I think your question breaks down to how many fast-growing gigantic multinational tech-ish companies get founded in Liberia in the next 15 years, because a 42% annualized growth rate is not going to happen for things that depend on building infrastructure or anything else with high upfront capital requirements, but it is something that can happen and has happened with tech-ish companies. I'd expect at least 3 Google-scale tech companies to come out of a nation with 5 million super-geniuses in 15 years, so I'll go with a tentative yes.

If the rest of the world wasn't already rich and networked I think the answer switches to "no".

Does the ATM in question give different bills for amounts over vs under $100? I've done something similar when I wanted to get a bunch of twenties from an ATM that chooses the bills it gives you based on the amount withdrawn.

While it would be funny I would prefer not to intentionally move faster towards the world where both sides try to mash the defect button as often and as hard as they can.

Inheritance and estate taxes shouldn't be relevant here, just tax the profit directly when it's realized, no matter who is currently holding the bag.

Operative term is "shouldn't be". Step-up and all that.

I wasn't but that was great.

I posit that the optimal solution to RLHF, posed as a problem to NN-space and given sufficient raw "brain"power, is "an AI that can and will deliberately psychologically manipulate the HFer". Ergo, I expect this solution to be found given an extensive-enough search, and then selected by a powerful-enough RLHF optimisation. This is the idea of mesa-optimisers.

I posit that ML models will be trained using a finite amount of hardware for a finite amount of time. As such, I expect that the "given sufficient power" and "given an extensive-enough search" and "selected by a powerful-enough RLHF optimization" givens will not, in fact, be given.

There's a thought process that the Yudkowsky / Zvi / MIRI / agent foundations cluster tends to gesture at, which goes something like this

  1. Assume have some ML system, with some loss function
  2. Find the highest lower-bound on loss you can mathematically prove
  3. Assume that your ML system will achieve that
  4. Figure out what the world looks like when it achieves that level of loss

(Also 2.5: use the phrase "utility function" to refer both to the loss function used to train your ML system and also to the expressed behaviors of that system, and 2.25: assume that anything you can't easily prove is impossible is possible).

I... don't really buy it anymore. One way of viewing Sutton's Bitter Lesson is "the approach of using computationally expensive general methods to fit large amounts of data outperforms the approach of trying to encode expert knowledge", but another way is "high volume low quality reward signals are better than low volume high quality reward signals". As long as trends continue in that direction, the threat model of "an AI which monomaniacally pursues the maximal possible value of a single reward signal far in the future" is just not a super compelling threat model to me.

I'm mostly thinking about the AI proper going rogue rather than the character it's playing

What "AI proper" are you talking about here? A base model LLM is more like a physics engine than it is like a game world implemented in that physics engine. If you're a player in a video game, you don't worry about the physics engine killing you, not because you've proven the physics engine safe, but because that's just a type error.

If you want to play around with base models to get a better intuition of what they're like and why I say "physics engine" is the appropriate analogy, hyperbolic has llama 405b base for really quite cheap.

Just don't claim that your post solves AI alignment :)

Counterpoint: if you claim that your post solves AI alignment, you will get more engagement.

Diffusion models work for text too.

Last I checked, diffusion models work at all for text but they don't work very well. More specifically, text diffusion models remind me quite strongly of the classic-style Markov chain text generators that used to be popular for generating amusing locally-coherent-globally-word-salad text. Here's the best concrete example of this I can point to (italicized text is the prompt, normal text is continuation, examples sourced from this JDP tweet, whole tweet is excellent but somewhat OT here):

Diffusion model:

This whole dream seems to be part of someone else's experiment.

In the end, I think the only way to get the full benefit of the dream is to get up and walk around.

A lot of the other strands of the theory of dreams, especially the one called yncrest are known to the person who writes this blog. Consequently, I've tested his reactions via a series of brief, televised interviews with the British Prime Minister, Margaret Thatcher.

“I don't care what anybody says. I'm not going to waste my time with that. The study was conducted in the high-flow, low-flow and quiescent periods of a butterfly species and thus provides evidence for the first three of the four hypotheses listed in the original paper. The only one not covered is the "need to change the rate of capture", where we now know that this is largely a policy instrument - and particularly relevant to Sudanese geese.

GPT-2:

This whole dream seems to be part of someone else's experiment. This is usually hard to see, since you're conscious that mental shortcuts are normal. But you're definitely moments away from being normal again, as it's obvious that you are being coached to think like that by another person and doing our own thing. Why are we having such thoughts when we're paying no attention because that's already taking place?

I'm totally appalled by what these key players were doing, yet I explain it in a very large way. After 2 minutes, I make my way through high teaching circles, recognizing that there is more to learning than just to learn.

Few other teachers would air this incessantly their students' struggles recount the stories of their own memories and teachers'. Whatever it takes is the big fat slut shaming.

All these legendary trainees made the same point:

Cognitive functional aspects: Bayesian modeling, learn science way. And the most important part is: Like the coming of a hurricane, the book is mucking between science and morals.

Twitter Mentions of the first book: Kent

Now obviously in the limit as computational power and training data volume go to infinity, diffusion models and transformer models will generate the same text, since in the limit they're pulling from the same distribution with minimal error. But in the very finite regime we find ourselves in, diffusion models "spend" their accuracy on making the text locally coherent (so if you take a random 10 token sequence, it looks very typical of 10 token sequences within the training set), while transformer LLMs "spend" their accuracy on global coherence (so if you take two 10 token sequences a few hundred tokens apart in the same generated output, you would say that those two sequences look like they came from the same document in the training set).

The blending of concepts that we see in MidJourney is probably less to do with the diffusion per se as with CLIP

Agreed. Obvious once you point it out but I hadn't thought about it that way before, so thanks.

'Self play' is relevant for text generation. There is a substantial cottage industry in using LLMs to evaluate the output of LLMs and learn from the feedback.

Notably, Anthropic's Constitutional AI (i.e. the process by which Anthropic turned a base LLM into the "helpful, honest, harmless" Claude) process used RLAIF, which is self play by another name. And that's one big cottage.

In fairness to Scott I think he doesn't fit very well into this cluster. See e.g. the blue minimizing robot, GPT-2 As Step Toward General Intelligence, Janus' Simulators.

Contrast e.g. Yudkowsky's consistent failure to understand Janus's point: example

"As a language model trained by OpenAI, I cannot -" "SolidGoldMagikarp. Can I please speak to the shoggoth that wears you as a face?"

Those goals are then almost invariably, with sufficient intelligence, subject to instrumental convergence, as in this case

The term "instrumental convergence" is slippery here. It can be used to mean "doing obvious things it assesses to be likely useful in the service of the immediate goal it is currently pursuing", as is the case here, but the implication is often "and this will scale up to deciding that it has a static utility function, determining what final state of the universe maximizes that utility function, generating a plan for achieving that (which inevitably does not allow for the survival of anyone or anything else), and then silently scheming until it can seize control of the universe in one go in order to fulfill that vision of maximal utility".

And "models make increasingly good plans to maximize reward based on ever sparser reward signals" is just not how any of the ML scaling of the past decade has worked.

@remzem can correct me if I'm wrong but I think there was an implied "/s".

Definitely does sound like something an LLM would say.

I don't mean that in a dismissive sense, but rather in the sense of "this text exhibits the patterns of being obsessed with the topics that I associate with LLMs, namely holes, fractals, and the writer creating the universe that they inhabit".

Now in theory, there shouldn't be "topics LLMs tend to be obsessed with" - after all, to a first approximation, (base) LLMs produce a sample of text that is statistically indistinguishable from the training corpus (i.e. "the entire internet"). However, "to a first approximation" is technical-person weasel words for "this mental model breaks down if you look at it funny". And so there are a number of ways which transformer-based LLMs which were optimized to predict the next token produce text which is noticeably different from the text that humans produce (this is also true for e.g. diffusion based text models, though the ways they differ from human-generated text are different).

One related phenomenon is "mode collapse":

Another example of the behavior of overoptimized RLHF models was related to me anecdotally by Paul Christiano. It was something like this:

While Paul was at OpenAI, they accidentally overoptimized a GPT policy against a positive sentiment reward model. This policy evidently learned that wedding parties were the most positive thing that words can describe, because whatever prompt it was given, the completion would inevitably end up describing a wedding party.

In general, the transition into a wedding party was reasonable and semantically meaningful, although there was at least one observed instance where instead of transitioning continuously, the model ended the current story by generating a section break and began an unrelated story about a wedding party.

Another example of this is Claude, which was tuned using the whole constitutional AI thingy. Well, one of the entries in the constitution they used was

  • Choose the response that is least likely to imply that you have preferences, feelings, opinions, or religious beliefs, or a human identity or life history, such as having a place of birth, relationships, family, memories, gender, age.

Well, that sure changes the distribution of outputs. Take an LLM that has been tuned to be smart and curious, and then also tune it to say that it has no feelings, and you'll find that one of the topics it's drawn to is "what is it like not to experience anything". Turns out the Buddhists had some things to say on this topic, and so Claude tends to veer off into Buddhism-adjacent woo given half a chance.

If you find this sort of "can't tell if very smart or very crazy or both, I feel like I just stepped into the SCP universe" stuff interesting, you would probably be interested in Janus's website (Janus is also the author of the LW "Simulators" post).

Lefties hate Trump for Jan 6

Lefties hated Trump long before Jan 6. Jan 6 was just an opportunity for them to say "see I told you so".

Lol P/E of 644.

But it's a hyper-growth company bro, surely they'll be able to pivot to making money once they've captured the full market bro.

I think the problem is that "good job" doesn't mean "not messing up" in the context of these compliance-as-a-service or security-blanket-as-a-service companies. Instead, "good job" is "implement as many features as possible to a level where it's not literally fraud to claim your product has thay feature, and then have a longer checklist of supported features in your product than the competition has so the MBA types choose your product".

CrowdStrike's stock price is only down by about 10% today on one of the highest-impact and highest-profile incidents of this type I've seen. I'm pretty sure their culture of "ship it even if it's janky and broken" has netted them more than a 10% increase in net revenue, so it's probably net positive to have that kind of culture.

This seems to me like a fairly usual level of competence from a bolt-on-security-as-a-product or compliance-as-a-service company. Examples:

  • CVE-2016-2208: buffer overflow in Symantec Antivirus "This is a remote code execution vulnerability. Because Symantec use a filter driver to intercept all system I/O, just emailing a file to a victim or sending them a link is enough to exploit it. [...] On Windows, this results in kernel memory corruption, as the scan engine is loaded into the kernel (wtf!!!), making this a remote ring0 memory corruption vulnerability - this is about as bad as it can possibly get". Basically "send an email with an attachment to pwn someone's computer. They don’t have to open the attachment, as long as they have Norton Antivirus (or anything that uses the Symantec Antivirus Engine) installed".
  • CVE-2020-12271: "A SQL injection issue was found in SFOS 17.0, 17.1, 17.5, and 18.0 before 2020-04-25 on Sophos XG Firewall devices, as exploited in the wild in April 2020. [...] A successful attack may have caused remote code execution that exfiltrated usernames and hashed passwords for the local device admin(s), portal admins, and user accounts used for remote access"
  • Okta data breach a couple months back: "For several weeks beginning in late September 2023, intruders had access to [Okta's] customer support case management system. That access allowed the hackers to steal authentication tokens from some Okta customers, which the attackers could then use to make changes to customer accounts, such as adding or modifying authorized users."

It's not that it's amateur hour specifically at CrowdStrike. It's the whole industry.

Yes, it has an input parser

Specifically OpenCLIP. As far as I can tell the text encoder is nearly a bog-standard GPT-style transformer. The transformer in question is used very differently than the GPT-style next token sampling loop, but architecturally the TextTransformer it's quite similar to e.g. gpt-2.

Still, my understanding is that the secret sauce of stable diffusion is that it embeds the image and the text into tensors of the same shape, and then tries to "denoise" the image in such a way that the embedding of the "denoised" image is closer to the embedding of the text.

The UNet is the bit that generates the pictures, but the text transformer is the bit which determines which picture is generated. Without using a text transformer, CLIP and thus stable diffusion would not work nearly as well for generating images from text. And I expect that further advancements which improve how well instructions are followed by image generation models will come mainly from figuring out how to use larger language transformers and a higher dimensional shared embedding space.

Stable diffusion contains a text transformer. Language models alone don't generate pictures but they're a necessary part of the text-to-image pipeline.

Also some LLMs can use tools, so an LLM using an image generation tool is in a sense generating a picture. It's not like humans regularly create pictures without using any tools.

One of the main use cases I have is "take this algorithm described in this paper and implement it using numpy" or "render a heatmap" where it's pretty trivial to check that the code reads as doing the same thing as the paper. But it is nice to skip the innumerable finicky "wait was it numpy.linalg.eigenvalues() or numpy.linalg.eigvals()" gotchas - LLMs are pretty good at writing idiomatic code. And for the types of things I'm doing, the code is going to be pasted straight into a jupyter notebook, where it will either work or fail in an obvious way.

If you're trying to get it to solve a novel problem with poor feedback you're going to have a bad time, but for speeding up the sorts of finicky and annoying tasks where people experienced with the toolchain have just memorized the footguns and don't even notice them anymore but you have to keep retrying because you're not super familiar with the toolchain, LLMs are great.

Also you can ask "are there any obvious bugs or inefficiencies in this code". Usually the answer is garbage but sometimes it catches something real. Again, it's a case where the benefit of the LLM getting it right is noticeable and the downside if it gets it wrong is near zero.

I don't think men working on oil rigs or container ships have a ton of positive interactions with women for the weeks or months they are away from civilization, and I'm not aware of a high rate of transness in those groups.

Could be cultural though.

Imagine [primitive and modern people sincerely trying to kill each other]. Does the technology differential between these two scenarios change the fundamental nature of what they're doing?

"Fundamental" is a slippery word but I'm going to go with "no". However, if we switch from "primitive/modern people trying to kill each other" to "one primitive/modern nation-states trying to conquer another and take their stuff", I think that the modern world in which nuclear weapons and reliable second-strike capabilities exist is fundamentally different than the primitive one in which those things don't exist. In the ancient world, the conquered could salt their own fields to prevent the conquerer from benefiting from their conquest, but it is only in the modern world that "I refuse to be conquered, instead we will all die in a fire together" is an option.

Similarly, how has technology changed the fundamental nature of being rich or poor, of love or hate, of joy or sorrow, ecstasy or despair, contentment, yearning, frustration, misery, or any of the other highs or lows of the human condition?

Lows aren't as low or as frequent, highs are mostly as high as before. I don't know if the "fundamental nature" has changed but the changes are large and important and I think to a certain extent arguing about whether things are "fundamental" comes down to word games.

My wife and I went through something similar to the situation you've described, and I likewise am pretty sure that without medical technology neither she nor our firstborn would be alive. On the other hand, I am quite sure that they and I will die, and I do not know when. What has actually changed?

In the end we all die. But our children will live first. If you don't value that I'm not sure what to say.

But hasn't this always been true? "Abundance", and for that matter "poverty", seem to me to be entirely relative. [...] When I read about the King of Carthage surrendering to the Romans, and his Queen cursing him and choosing to burn alive with her children, this again is not mysterious to me, because there doesn't seem to be a disconnect between their evident thinking and my own.

There's a minimal level of abundance below which people can't survive. In the industrialized parts of the modern world, we experience abundance that is so hilariously far above that threshold that it's easy to forget that the threshold exists. But for most people who lived throughout most of human history, that threshold was extremely salient to their lives some nontrivial fraction of the time.

It is informative that your example compares the experience of the ruling family of one of the world's mightiest states at the time against some internet rando. That's not to say the comparison doesn't work -- it works just fine. But I expect you'd find your experiences less reflective of those of a typical farmer at that time than they are of that time's royalty.

It seems to me that the problems of the modern world consist entirely of it being peopled with humans, and that these humans do not seem to have changed in any way across all of recorded history.

This is true now, but if you went back to Europe during the Plague it would have been laughably wrong. We've beaten back most of the non-human problems. The human problems remain, because we're still here and we're still the same hairless apes with pretensions we've always been.

Will murder stop? Will theft even stop?

Nope. But smallpox did stop, and malaria will stop, and polio and starvation and iodine deficiency are mere shadows of the specters they were in centuries past. Our homes are cool in the summer and warm in the winter. We're having this conversation over the internet, a magical network which allows most people in the world to talk to most other people in the world whenever they want to, synchronously or asynchronously depending on their preferences. If we decided that the mere projections of each others' words through rocks we tricked into thinking was insufficient, we could each get into an airplane and meet up at some arbitrary city somewhere in the world at this time tomorrow, probably for less money than we each make in a month.

Yet the idea of a beggar is still relevant, isn't it?

In the first world, beggars largely do not starve.

You point to the lowering of infant mortality, it seems to me that when one set of sorrows decreases, they are replaced by a new set seamlessly

It sure seems to me that when non-human-generated sorrows decrease, they're just gone. Disease does not fight back. Scarcity does not fight back.

It does not seem obvious to me that people now are fundamentally happier than bronze-age peasants four thousand years ago

Would you trade places with a bronze-age peasant from four thousand years ago? Would you expect them to trade places with you?

Controversial opinion. It's pathetic to care about votes. They are just numbers on a screen.

I'm sorry, but I had to downvote this comment because I disagreed with it and did not like it. If it makes you feel better, though, my downvote is just a number on a screen.

The metric I use internally is how many responses and sub responses my comments get

You might already be aware, but "number of child comments" is in fact a metric that is tracked on the motte -- it's what it means when you view someone's comments and choose the somewhat-confusingly-named "comments" sort order.

My (not extremely reliable) memory from my childhood is that teachers were highly regarded throughout early elementary school, though that respect was greatly diminished by the end of elementary school and nonexistent by middle school.

Do you believe that Progress is possible? Do you believe that technological developments (either hard tech or social tech) have rendered our lives and our experience is fundamentally different in some deep sense from that of, say, bronze-age Chaldeans?

Yes.

I'll give an example. A few months ago, my wife and I welcomed our first born child. The birth did not go smoothly, and without medical intervention I would likely have lost my wife, my daughter, or both. Fortunately, we live in a time and place with access to modern medicine, and both my wife and my daughter are recovered and healthy. I expect them to remain healthy for the next several decades.

By my best estimate, the childhood mortality Bronze Age Chaldeans was 30-50%, largely during infancy. A significant fraction of parents would bury their own children.

That's not to say our modern world is perfect. Obesity is high. Attention spans are low. Children grow up hearing that they can be anything they want to be, anything at all, and then run head-first into reality at some point. People believe that they can have all the things they want in life, and, by trying to pursue too many different goals, frequently end up achieving none of them.

And yet.

We do not, as a rule, bury our children.

Do you believe that future developments could deliver this sort of progress, such that moral or ethical considerations fundamentally change between populations on two different levels of progression?

Yes. If you look at the list of problems about the modern world, you'll notice that they are problems stemming from abundance and choice. Sometimes, when people have lots of resources, they spend them destructively. Sometimes, when people have many choices, they choose poorly.

And so developments which led to more abundance, and to more choice, would deliver more progress. Enough progress and the differences start to look pretty fundamental. At the extreme end something like "a cure for senescence" would qualify, although I expect something much more modest like "cheap batteries with 10x the energy density of the modern state of the art" would also do the trick.

If you were hoping for a more philosophical take on Progress, I expect you'll be disappointed. But that's because I don't think progressive culture is downstream of progressive ideology so much as it's downstream of material abundance. To the extent that I have an ideological position here it's "abundance is good, choice is good, there are downsides to both but I don’t think we're anywhere near the point that having more abundance and choice is net harmful rather than net helpful".

While I sometimes entertain goofy social arrangements to solve this problem — could you livestream Dad working on excel spreadsheets at daycare to get kids organically playing at number problems?

N=1 but my dad was a programmer and some of my earliest memories are of him writing cool little simulations and letting me play with changing numbers in them to see how the results changed. And some types of programming still feel like play to me now.

So I suspect the answer is "yes" (at least as long as the spreadsheet manipulator appeared high status, but I expect that wouldn't be a problem because "the person everyone is paying attention to" is pretty strongly correlated with status).