This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
FOSS and The XZ Problem
Security Boulevard reports:
The exploit would allow remote code execution as root in a wide majority of systemd-based Linux (and Mac OSX, thanks homebrew!) machines. There's some reasonable complaints that some CVE ratings are prone to inflation, but this has absolutely earned a 10/10, would not recommend. Thankfully, this was caught before the full exploit made it to many fixed release Linux distros, and most rolling-release distros either would not have updated so quickly or would not yet be vulnerable (and, presumably, will be updating to fixed versions of XZ quickly), with the exception of a handful of rarely-used Debian options. Uh, for the stuff that's been caught so far.
Summary and FAQ, for the more technically minded reader, the NIST CVE is here, background of initial discovery at here.
Ok, most of us who'd care remember Heartbleed. What's different here?
In this case, the exploit was near-certainly introduced intentionally by a co-maintainer of the library XZ Utils, by smuggling code into a binary test file, months apart from adding calls to execute that test file from live environments, and then working to hide any evidence. The combination of complexity in the attack (requiring fairly deep knowledge of a wide variety of Linux internals) and bizarreness of exploit steps (his FOSS history is sprinkled with a replacing safe functions with their unsafe precursors, or adding loose periods in cmake files) leaves nearly zero chance that this is unintentional, and the guy has since disappeared. He was boosted into co-maintainership only recently, and only after the original maintainer was pressured to pick him up by a strangely large barrage of very picky users. The author even pushed to have these updates shoved into Fedora early.
Most mainstream technical advisories aren't outright calling this a nation-state actor, but The Grugq is pretty willing to describe whoever did it as an 'intelligence agency', whether government or private, and with cause. Both the amount of effort and time put into this attack is vast, and the scope of vulnerability it produced extreme -- though this might be the 'cope' answer, since an individual or small-private-group running this level of complex attack is even more disturbing. It's paranoid to start wondering how much of the discussion aimed encouraging XZ's maintainer to take on the bad actor here as a co-maintainer, but as people are having more and more trouble finding evidence of their existence since, it might not be paranoid enough.
There's a lot of potential takeaways:
The Many Eyes theory of software development worked. This was an incredibly subtle attack that few developers would have been able to catch, by an adversary willing to put years into developing trust and sneaking exploit in piecemeal.
Except it was caught because a Microsoft (Postgres!) developer, without looking at the code, noticed a performance impact. Shit.
This attack heavily exploited access through the FOSS community: the author was able to join sight-unseen through a year of purely digital communications, and the 'business decision' of co-maintainership came through a lot of pressure from randos or anons.
Except that's something that can happen in corporate or government environments, too. There are places where every prospective employee gets a full background check and a free prostate exam, but they're the outlier even for dotmil spheres. Many employers are having trouble verifying that prospective recruits can even code, and most tech companies openly welcome recent immigrants or international workers that would be hard to investigate at best. Maybe they would have recognized that the guy with a stereotypical Indian name didn't talk like a native Indian, but I wouldn't bet on even that. And then there's just the stupid stuff that doesn't have to involve employees at all.
The attack space is big, and probably bigger than it needs to be. The old school of thought was that you'd only 'really' need to do a serious security audit of services actually being exposed, and perhaps some specialty stuff like firewall software, but people are going to be spending months looking for weird calls in any software run in privileged modes. One of many
boneheadedcontroversial bits of systemd was the increased reliance on outside libraries compared to precursors like SysV Init. While some people do pass tar.xz around, XZ's main use in systemd seems to be related to loading replacement keys or VMs, and it's not quite clear exactly why that's something that needs to be baked into systemd directly.But a compression library seems just after cryptographic libraries are a reasonable thing to not roll your own, and even if this particular use for this particular library might have been avoidable, you're probably not going to be able to trim that much out, and you might not even be able to trim this.
There's a lot of this that seems like the chickens coming home to roost for bad practices in FOSS development: random test binary blobs ending up on user systems, build systems that either fail-silently on hard-to-notice errors or spam so much random text no one looks at it, building from tarballs, so on.
But getting rid of bad or lazy dev practices seems one of those things that's just not gonna happen.
The attacker was able to get a lot of trust so quickly because significant part of modern digital infrastructure depended on a library no one cared about. The various requests for XZ updates and co-maintainer permissions look so bizarre because in a library that does one small thing very well, it's quite possible only attackers cared. 7Zip is everywhere in the Windows world, but even a lot of IT people don't know who makes it (Igor Patlov?).
But there's a lot of these dependencies, and it's not clear that level of trust was necessary -- quite a lot of maintainers wouldn't have caught this sort of indirect attack, and no small part of the exploit depended on behavior introduced to libraries that were 'well'-maintained. Detecting novel attacks at all is a messy field at best, and this sort of distributed attack might not be possible to detect at the library level even in theory.
And there's far more varied attack spaces available than just waiting for a lead dev to burn out. I'm a big fan of pointing out how much cash Google is willing to throw around for a more visible sort of ownage of Mozilla and the Raspberry Pi Foundation, but the full breadth of the FOSS world runs on a shoestring budget for how much of the world depends on it working and working well. In theory, reputation is supposed to cover the gap, and a dev with a great GitHub commit history can name their price. In practice, the previous maintainer of XZ was working on XZ for Java, and you haven't heard of Lasse Collin (and may not even recognize xz as a file extension!).
((For culture war bonus points, I can think of a way to excise original maintainers so hard that their co-maintainers have their employment threatened.))
There's been calls for some sort of big-business-sponsored security audits, and as annoying as the politics of that get, there's a not-unreasonable point that they should really want to do that. This particular exploit had some code to stop it from running on Google servers (maybe to slow recognition?), but there's a ton of big businesses that would have been in deep shit had it not been recognized. "If everyone's responsible, no one is", but neither the SEC nor ransomware devs care if you're responsible.
But the punchline to the Google's funding of various FOSS (or not-quite-F-or-O, like RaspberryPi) groups is that even the best-funded groups aren't doing that hot, for even the most trivial problem. Canonical is one of the better-funded groups, and it's gotten them into a variety of places (default for WSL!) and they can't bother to maintain manual review for new Snaps despite years of hilariously bad malware.
But it's not clear that it's reasonable or possible to actually audit the critical stuff; it's easier to write code than to seriously audit it, and we're not just a little shy on audit capabilities, but orders of magnitude too low.
It's unlikely this is the first time something like this has happened. TheGrugq is professionally paranoid and notes that this looks like bad luck, and that strikes me more as cautious than pessimistic.
My takeaways:
Some more comments on the OP:
I think there is a big difference. Rolling out your own crypto is a big no-no because they are hard to get right, and any mistakes likely leave you vulnerable.
Rolling out your own compression is much less evil: there is certainly some potential for arbitrary code execution vulnerabilities, but not more than with handling any other file parsing. With regard to generally reinventing the wheel versus loading wheels from a zillion different libraries, each of them with their own dependency chains, there is probably some reasonable middle ground. For something like sshd which sits on a security boundary, the obvious way in retrospect to add systemd logging would be to implement the interface from the scratch instead of including a bloated libsystemd.
Data point: As some casual linux user, I recognize the xz file extension. Before last week, the main thing I could have told you about it was that it was a compression commonly used for tar files, the third one I am aware of after gz and bz2. GNU tar wants -J when handling xz. I would have guessed that the fact that it de-facto replaced bz2 is likely due to the fact that it is better on at least some metrics, but have no clue how the xz algorithm works in particular.
On the plus side, the fact that the attackers stayed in userspace instead of having /usr/bin/sshd load some kernel model seems to indicate that a stealthy compromise of the kernel is hard? Yay for NSA's SELinux?
I like to think that this will get better as time goes on. If you think about it, humans have only really been writing software at an industrial scale for two, maybe three decades now. We're not good at it yet.
Every single one of us is running a kernel that was written in the 90s using paradigms formed in the 80s with a computer language that was invented in the 70s.
So little about how we do computing has even caught up to modern thinking. I don't know if Rust specifically is the future, but something like it is.
I’ve heard it said that Rust is the new C++ but Zig is the new C.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link