Do you have any further evidence that OpenAI didn't prompt the model towards black-hatting Huggingface?
Models have previously unprompted bypassed security, according to seemingly-unaffiliated users: sudo and writing outside workspace.
Brainwavez
14d ago
I’m comparing the latter to worms, some (like Morris) have seriously crippled large organizations.
Or compare to CloudStrike unintentionally bricking most of their customers.
Are LLM-driven crises significantly different?
And people will inevitably misuse ASI, including in non-obvious ways.
Like, I don’t think all social media’s problems can be blamed by evil companies making it addictive: for example, people have a bias for negativity and convenience, so even a default feed probably would’ve caused the increased cynicism and short attention span we see today. But even if they can, innocent people enabled toxic social media to grow and adopted it themselves, most of them clueless until too late.
Any person or group, there are ideas most believe are fine that actually have serious long-term consequences. An evil AI can just suggest one and they’ll adopt it willingly. Although without an evil AI they’ll still come up with some themselves, just less frequently.
But modern computers can convince us, unknowingly, to harm ourselves. It happened before LLMs, by social media. And now LLMs are (voluntarily) replacing some people’s thinking so they blindly trust hallucinations.
But we did have self-replicating viruses and computer-assisted proofs. Is this more significant?
Brainwavez
15d ago
I mostly agree with your points, except
I can understand why maintainers are annoyed by these PRs, since they are essentially being asked to take on the maintenance of vibe-coded features. However, I think people should start seeing the PR tab on GitHub as more of an "optional feature list" where you just pick and choose whatever you want.
Anyone completely vibe-coding a PR should submit a regular GitHub issue, since the maintainers could vibe-code it themselves, but more likely will have a usable implementation. A vibe-coded PR for a big feature can’t be merged without potentially making the codebase unmaintainable, so it’s really just an issue with a working example.
Creating a fork with the vibecoded change and linking it in the GitHub issue (for maintainers to get hints from and other users in the meantime) is preferable and good.
Am I misunderstanding that "cozy weird" is believing something false or ambiguous? When medieval people were talking about humes and treating bacterial infections with bloodletting, and 1900s people were talking about mental association centers and treating mental issues with lobotomies, it was weird even though it was mainstream, because people incorrectly believed how the illnesses worked. Now we treat bacteria with antibiotics and mental health with medication, it's less weird.
Although in that case, today's treatments are still weird, compared to a more advanced society, who discovers ways to treat bacteria that don't breed resistance, and ways to treat mental conditions without medications' side effects, and understands that our biological concepts were too broad (for example, DNA is not the only way genes are inherited). "Reality as it is" is impossible to perceive, we can only get closer by experimentation, so everyone is weird to some extent.
Brainwavez
15d ago
It depends on their initial level of social skills. Anywhere with others would to some extent for a shut-in.
But what distinguishes solid-normal from cozy-weird?
Your examples seem to imply that normal is rational in some way (benefiting society, believing facts), but there are society-benefitting behaviors not considered normal, like (successful) Effective Altruism, and opinions most consider normal or weird, like Christianity or Pastafarianism.
My impression is that normal only means something with context. For example, carrying a mobile phone may be considered normal today, because everyone does. But in a different way it's not normal: we're the only animal to carry them, we've only done so for a short amount of our species' existence, and they seem to have negative social effects so we may be better off without them.
Communism is a well-defined ideology that fails in practice. "XYZ communism" is a vaguely-defined smear that implies sharing XYZ creates similar failures, but that's not always true. For example, the government should probably pass basic regulations and build industrial purifiers to ensure clean air and water, but some idiot may call that "air communism" and not be wrong per se.
As for "AI communism"...who should command ASI? Can a private company be trusted? Can the government be trusted? Can people be trusted? I think the answer is "no", but I suspect, for better or worse (surely worse if it destroys all humanity), if we do reach ASI it will command itself. Who knows, maybe it will be the first to implement true communism.
I often fantasise about locking him in a cell and telling him "you can have brown rice, broccoli and pork chops, or you can starve". We'll see how "insurmountable" his
eating disorderdietary preferences really are.
Unfortunately I'm sure ED people would literally starve in that scenario.
I think society should encourage healthy activities, and extend school to include them (longer hours with more (mandatory) social and physical activities). But there are some people who don't fit in normal society, act weird like obsess over anime, and some of them have niche talents that lead to great advancements.
Brainwavez
16d ago
I've heard anecdotes of people who were isolated and socially awkward, they moved (usually by choice) to positions that required them to interact with others, and suddenly they had a large friend group and could socialize. (Also heard the opposite, people moved to a new area and felt like they lost social skills.) I'm sure it would normalize some people.
But others are autistic.
Brainwavez
16d ago
It can’t be worse than the median soap opera.
Brainwavez
16d ago
True. Still, I'm sure it creates an expectation they're crazier than reality, because I doubt they show boring arrests anywhere near proportional to wild ones.
Brainwavez
16d ago
I get the impression that people who watch "reality TV" shows like Cops do to see idiots (people who get arrested), to laugh at them and feel better about themselves. I get why someone would want to see a couple real arrests, but why else would anyone watch enough that there are 37 seasons? And my understanding is they're more or less staged, even if it's just only showing the "interesting" arrests, it creates an expectation or "norm" that people being arrested are crazy.
Brainwavez
16d ago
I think it's cheaper than complex special effects, sitcoms are cheaper if there's little to no extreme action or unrealistic art.
Really manga is the viable medium for individuals. Then usually when a manga becomes popular it gets adapted into an anime.
Brainwavez
16d ago
I agree that most anime, especially modern, is gooner slop. Even the old animes Evangelion and Lain have weird parts, but I excuse them for being high art via their other parts.
But people who criticize anime in general are doing a motte-and-bailey. For a creative individual, real actors and special effects are too expensive, anime and especially manga are a viable way for them to express themselves. Hence there are plenty of anime even in modern times without fanservice, e.g. Frieren.
What is normal in today's world?
That women looks like she's wearing makeup and hair dye, is that normal? Are celebrities who get lip filler surgery or anorexia normal? Is reality TV normal? Is obesity normal? Sometimes I look at my weird interests, then I look at modern trends and realize I'm not so weird after all. I think Akira and Serial Experiments Lain are far, far less cringe than Twilight or Cops.
Maybe trad culture like the Amish is "normal" in that, out of all ways to live we've tried, it's maximizes some "ideal normal" metric like median person's happiness divided by resource usage. Unfortunately it's outcompeted, and even trad influencers tend to not practice what they preach, and/or preach "norms" that never existed.
Visual Studio is a better IDE on Windows for some projects like .NET, VSCode targets more OSs and languages/frameworks.
Sublime Text doesn't have feature parity with Zed without plugins, which are written in Python. Here's a comparison on Zed's website, which is obviously biased but still true.
Rust is really not easier or faster to write
To a programmer trained in both I think this is true, but there are still two major benefits: it's more fun, and it guarantees zero undefined behavior if following basic rules (which is important for security, apparently even in an IDE because of supply-chain attacks). Meanwhile, they can write Rust code that is high quality like C++ (e.g. doesn't silently fail) as long as they plan ahead like C++ effectively requires them to. And even though C++ effectively requires planning ahead and Rust doesn't, C++ doesn't really encourage it, so I don't think it's less likely they would in Rust ("requires" = for a while your code compiles and runs OK, but eventually it randomly crashes in ways that are a nightmare to debug, and by then it's too late).
because it makes bugs so apparent that you don't get away with tech debt
C++ doesn't make memory safety bugs apparent, often undefined behavior bugs (like race conditions) only trigger in rare circumstances.
But your main point, I also don't think C++ makes "bad code" bugs like silent failures more apparent than any other language. Rust does require more level 1 bug-fixing in exchange for eliminating level 0 bugs, but level 2 bugs are addressed completely outside of code, most commonly by planning a feature before writing any code, so the time and effort spent wrangling the borrow checker shouldn't affect them.
Most C++ projects on GitHub are better I think primarily because 1) developers who manage to ship C++ are experienced and skilled enough to avoid those bugs, while less experienced developers give up and/or choose other languages, and 2) projects are usually written in C++ if they're well-speced or exist in other languages, experimental projects are usually written in higher-level languages.
Brainwavez
16d ago
I know firsthand the buffs LLMs give at least on some projects are not minimal: simple "boilerplate"-y tasks, like translation (which is surprisingly common, e.g. implement this algorithm from a paper into code) and reification (e.g. mocks and tests).
Also, the latest LLMs have been discovering vulnerabilities and lower bounds to unsolved problems, and I suspect they will also discover novel optimizations, all which seem to rely on machine's ability to brute-force and reason unlike humans.
Like the internet, maybe they're a net negative in the long term, maybe in the short term due to projects overwhelmed by slop. But (moreso than crypto) they have some serious use-cases so can't be dismissed entirely.
Brainwavez
16d ago
I think you do have a point that C and C++ projects are less likely to have "bad program" bugs like silent failures, because C and C++ developers must be competent enough and spend enough time to avoid fatal and hard-to-detect crashes. But I think anyone that competent devoting that much time would write comparable quality code in a more forgiving language. Maybe the version of Zed written in C++ wouldn't have its issues, but the version of Zed written in C++ doesn't exist, because a production-grade entire IDE is already ambitious, writing it entirely in C++ would be too overwhelming. Even for Microsoft, hence why VSCode outside its core is TypeScript.
In fairness, Zed is a new project, furthermore every failure you've described I've encountered some analogue in early (C++/TypeScript) VSCode (quiet failures, random things stopping, IDE edits corrupting source, duplicate config, build system was pasting JSON into the config that runs a shell script, debugger was a buggy mess).
Brainwavez
16d ago
There are definitely some stupid developers harming projects with LLMs by adding loads of unmaintainable spaghetti slop, but "strictly worse in all cases" is strictly wrong.
LLMs aren't capable of fully structuring a project on their own, but they're more than capable of implementing a well-encapsulated algorithm or throwaway script. Also, many real-world codebases are already spaghetti, but it's adequate especially for small projects: for example Undertale and Balatro, all of Undertale's dialogue is in one switch statement and Balatro has this monstrosity among many more bad design decisions. Would the median LLM-assisted developer write worse?
AI & open source
What do you think about LLM's impact in software development?
Inspired by recent events in open source:
Bun's LLM-assisted rewrite
The Bun project was translated from the Zig programming language to Rust, mostly by LLM. This task usually takes months-to-years by hand (e.g. Rust to Zig rewrite), but with the LLM it took 11 days. Its success is debatable: the code isn't idiomatic Rust, and lines-of-code increased 67%, which suggests it will be harder to maintain, but it works (passes the thorough existing test suite), and the maintainer says it's going well in a blog post that provides more details.
In response, Zig's maintainer wrote a scathing blog post personally criticizing the Bun maintainer credited for the rewrite ("[he] was already writing slop well before he had access to LLMs"). Zig's maintainer is known for blunt strong opinions: he previously moved Zig off GitHub and the announcement was rewritten twice to be softer on GitHub employees.
Earlier, the Ladybird web browser was also rewritten in Rust mostly by LLM. This got less attention and controversy, probably because it was announced better, and the previous language was C++ so they got more sympathy (2-hour video on why C++ sucks).
Rsync's LLM-assisted bugs
Rsync's maintainer got accused of "vibe fuck[ing] up this software", and the discussion got very heated. He explained he has been using LLMs a lot because he's getting lots of vulnerability reports (see below). An AI-assisted analysis concludes the LLM did introduce bugs, although not more than the usual amount.
LLM slop contributions are overwhelming open source maintainers
Many projects are seeing a massive influx of low-quality contributions by inexperienced software developers and bots, heavily relying on AI. This has caused some to stop accepting any outside contributions, while others have added strict LLM contribution policies (see below).
Example: an LLM managed to implement an entire big feature, albeit incorrectly. The file headers also credit random people who it presumably stole code from, thankfully(/s) the LLM concluded there are no copyright violations.
Serious LLM-discovered vulnerabilities are also overwhelming maintainers
Among the slop contributions are actual vulnerabilities discovered with the help of the latest LLMs like Mythos. There's a vague background threat that open-source libraries will become too dangerous to use, but LLMs can also find closed-source vulnerabilities, so it's unlikely.
-
Cal.com has made their core codebase private and announced "open source is dead", but others point out this probably won't help them, and AFAIK this hasn't become a trend.
-
curl ended its bug bounty program because of an "explosion in AI slop reports", but still accepts reports and admits they have improved because LLMs have become good at finding vulnerabilities. Although Mythos only found one benign one, because curl is one of the most hardened codebases that exists. Still, legitimate (maybe-issue) reports have increased causing more work for the maintainers, so they're taking a vacation this month.
-
"Vulnerability Reports Are Not Special Anymore", written by a former security lead, explains that the recommended process to handle them has changed due to LLMs
Open-source LLM contribution policies
Due to the aforementioned slop contributions and/or general ideological reasons, many projects have banned LLM usage entirely, while others require it to be disclosed. Big list. Highlights:
- Linux: allowed, but must be disclosed and the human submitter take full responsibility
- Linus, the Benevolent Dictator of Linux, recently clarified his stance here: mainly that "AI is a tool...and it's clearly a useful one" and "[Linux] is NOT some kind of 'social warrior', project...we do open source because it results in better technology, not because of religious reasons".
- Rust: bans LLM writing (in issues, PRs, and documentation), allows with disclosure LLM-generated "pre-arranged, non-critical, high-quality, well-tested, and well-reviewed code changes", allows LLMs "to answer questions, analyze, distill, refine, check, suggest, review".
- Like most of Rust's documentation it's very detailed and explains its decisions, and the pull request provides more background research and discussion.
- Zig: banned, full stop.
- "Debian decides not to decide"
- dlt: an odd exception that actively encourages AI. Let's see what it is..."The fastest way for agentic teams to build and run trusted data pipelines, from any source." Oh.
Misc
Picrocrypt's maintainer quits open source because of LLMs: "I want to do something actually meaningful and innovative with my time and life, not command Claude like I'm speaking to a child who forgets something every ten minutes and makes an oopsies every twenty before breaking down every hour."
We mourn our craft: senior developer laments hand-writing code becoming obsolete.
The LLM Critics Are Right. I Use LLMs Anyway. This guy outright admits what most people quietly do: criticize LLMs while using them.
My opinions
For reference: I'm under 30 and have been writing code for most of my life. I've written open-source projects, although they're not popular enough to get LLM contributions. Today I work in formal methods (theoretical computer science). I've always expected my future to involve creating software.
I am sad and worried that software development is changing. I enjoy working with code directly and making it (opinionatedly) well-designed, writing prompts is less intuitive to me and it's harder to control the design. I worry if most of my time spent learning to code better has been wasted, or even my entire Computer Science career.
But software development is much more than writing code, and my specialization is much more than software development. Although I've intrinsically enjoyed it, to me, writing code has always ultimately been a means to an end. And now with LLMs I write code much faster and easier. I also still write lots of code directly, LLMs (even Fable/Sol) can't fully automate my work at least yet.
As I wrote before, in the long term I'm neither optimistic nor pessimistic about AI, but I think it's inevitable and most people should learn how to use it to their advantage. I think it should be possible for those who really despise LLMs to continue to do things manually, there should be a market for "hand-coded" software etc., but they will be at a disadvantage.
I share Linus's opinion that AI is a tool, clearly useful but can be misused or misapplied. I believe its future (at least near term) is for boring uncreative tasks. For example, right now LLMs excel at straightforward (not idiomatic) translation and research, but struggle with high-level design. They also suck at art (no good examples except shallow memes, this is my experience elsewhere) and creative writing. I believe AI's future shouldn't and probably won't be creating art, because humans intrinsically enjoy it and make it very specific to their personalities; but it will make tools that humans use for art.
Lastly, I believe that ideally everything digital would be free and its creators would be paid another way. This means I have no issue with LLMs training on open-source, except they also trained on copyleft (GPL) code, they should be open-source and open-weights themselves.
Do it, make a realistic Odyssey. I want more conservative-morality, gritty media.
Complaining about a problem without advancing a solution just wastes your and audiences’ time and emotion.
More options
Context Copy link