site banner

Culture War Roundup for the week of July 15, 2024

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

9
Jump in the discussion.

No email address required.

Ok this might just be funny to me, but the CloudStrike Crowdstrike worldwide outage is the funniest thing to happen in computer security this decade.

If you haven't caught up, 100+ million (billion?) computers around the world were simulatenously broken in an instant. It's black comedy for sure. Hospital & emergency systems around the world have crawled to a halt, and there will be a few hundred deaths that will be traced back to this event. Millions of $$ will be lost. But, the humor comes from the cause of it.

Here is how things panned out:

  • CloudStrike Crowdstrike is a 100 billion valuation tech company that provides security services to a bulk of the world business.
  • Most sensitive organizations (govt, military, healthcare) will refuse to work with you unless you are compliant & all your machines have this installed.
  • It is effectively an anti-virus that sits 1 level below your operating system, 'protecting' your organization from 'bad outcomes'.
  • On Friday afternoon (which we all know is the best time), CloudStrike Crowdstrike deployed a software update that began this outage
  • For any other software this would be a simple restart or uninstall away, but since CloudStrike Crowdstrike is a 'trusted' secuirty tool, it sits under the OS layer, bricking the whole device.
  • Alright, so how do they fix it ?...... THEY CANT !
  • The beauty of bricked device, is you can't send any more software updates to it. You must do it manually. Raw dog it like the 90s.....all 100 million of these computers.
  • That's bad, but surely they can give those instructions to people and each person can fix their laptops themselves. Divide the labor.....
  • NOPE !
  • This software is used in vending machines, kiosks, tablet displays....and all sorts of devices that sometimes don't have keyboards and other times haven't been looked at for years. But at least there is a fix right ?
  • Yes....... but it needs you to start the computer in safe mode....which you can't because 'Bitlocker'.
  • Ah yes, Bitlocker. Turns out, another security measure, makes it so that 99% of a company's employees can't open safe mode.
  • So yes, a few hundred IT people will be responsible for fixing hundreds upon hundreds of laptops, daily, for weeks !

This is the Y2k that was promised.

The world spends billions in computer security every year, and no virus has managed the kind of world-wide disruption caused by one simple bug by the premier security company in the world.


No direct culture war implications, but goes to show just how much of a house-of-cards the tech ecosystem is. 1 little, simple, stupid bug can bring the whole world to a halt. Yet, the industry continues quarterly-earnings chasing.

Jobs keep getting cut, senior members get aged out, timelines get thinner and 'how many features did you deploy' remains the only metric for evaluation.

In tech, staying at a job for more than 3 years is seen as coasting. Devs are increasingly expected to do everything, because 'everyone should be full stack' and everything that isn't feature development (testing, staging, canaries) get deprioritized. Overworked novices means carelessness, carelessness creates mistakes.

At the same time, devs get zero agency. Random HR types make list of regulations mandating certain checkboxes for compliance, while having near-zero knowledge of the risks-and-benefits of these technical decisions. Therefore, the implications of a mistake are opaque to decisions makers. So by being compliant, you've suddenly given CloudStrike Crowdstrike a button to shut your entire business down.

This kind of error should literally be impossible in a company of the size of CloudStrike Crowdstrike . If such an error happens, it should be impossible for giant corporations to crumble zero backup. Incompetence on display, on all sides. Having worked in 'prestigious tech companies', especially in 2024, it isn't surprising. At times, the internal dysfunction is seriously alarming, other times it's a tuesday.


I'm not going to hope for much out of this. Just like Spectre & Solar , people will cry about it for weeks, demand change and everyone will get collective amnesia about it as the next quarter rolls around.

End of the day, tech workers are treated as disposable labor. Executive bean counters are divorced from the product. And the stock price is the only incentive that matters.

As long as tech is run by MBAs and smooth talkers, this will go on.

Some choice photos:

The competency crisis rages on. Boeing's planes fall out of the sky. The Secret Service forgets to check the nearby roof. Anti-virus software bricks your computer. These sorts of incidents have always happened, but it's hard to deny that they have gotten more frequent.

Boeing used to be better. I believe the Secret Service was as well. But anti-virus software, and the companies which make it, have always sucked.

Ehh I'm going to press X to doubt on the Secret service.

John F. Kennedy got Killed (I'll admit Trump would have been killed by Lee Harvey Oswald too)

Gerald Ford had 2 assasination attempts on him both of which he got lucky and survived but both were even crazier than Trumps

and just looking through wikipedia the list is just so long and full of examples that it beggars the question if Trump was even remotely unusual.

Boeing I'll grant you though, I think a part of it is that every corporation has its ups and downs and we have 1 down for Boeing right now, but remember the ford pinto? Boeing's issues are nowhere near as bad.

Boeing's planes fall out of the sky.

I could be wrong, but the number of fatal Boeing crashes or lesser incidents is not an outlier compared to past incidents and other manufactures before all the media scrutiny. Anyone remember the 737 rudder jams during the 90s? https://en.wikipedia.org/wiki/Boeing_737_rudder_issues#:~:text=During%20the%201990s%2C%20a%20series,board%2C%20157%20people%20in%20total.

It was a different model and hardly got similar media attention despite two major accidents with lots of fatalities close together

The Boeing issue was somwhat unique in that it was arguably the result of a vulnerability that had been purposefully introduced.

A conscious choice was made to change the emergancy autopilot disconnect from a physical switch to a software one and also to exempt certain autopilot functions from said disconnect switch thus invalidating the existing pilot checklist procedure for bad air data.

but it's hard to deny that they have gotten more frequent.

I'm always skeptical but never dismissive of such common sense. It could be recency bias and the availability heuristic at work.

I am starting to think there's the opposite of that kind of bias at play. 'Instinct distrust bias'?

I don't know what to call it, but it certainly feels like a lot of people turn very 'skeptical' when an aspect of their supported or preferred worldview is poked at in some way. The most obvious example of this would be mass immigration and the rise of housing prices. Implying a causal connection simply isn't a part of the program. Yet instinct would tell us it's the most obvious and important part of the entire problem in most if not all western countries.

Ditto depressed wages, rise of "the gig economy", etc...

Pick me! I’ll deny it!

I have zero reason to believe capability has gotten worse by any reasonable metric. Maybe—just maybe—that’s propped up by technology even as competency has tanked? But if so, I think there should be better evidence than black swans.

Compare complaints about the land boats of old. Why can’t we buy sweet Caddys anymore? I dunno, because they were death traps in an accident.

I’m still trying to find that Onion skit about accidentally invading the wrong Middle Eastern country.

Why can’t we buy sweet Caddys anymore? I dunno, because they were death traps in an accident.

Well, I think it has more to do with fuel efficiency standards. They were also death traps, or not as perfectly safe as possible, but rounding off all the edges for aerodynamic efficiency gives all calls a sameness that's striking when compared to older designs.

You can build a land boat that's as safe as you like, but it's not going to meet fuel efficiency standards unless it's classified as a truck somehow. This also relates to the rise of SUVs: they're not-sedans, and so they don't have the same standards.

I remember a video about the old standard of round headlights. Super convenient for everything except aerodynamics. There was an awkward transition where companies tried to put the aero shell around their legally-mandated headlights, but that was unnecessary after the regulation got removed. Wish I could find it again.

I agree that our competence probably hasn’t declined that much. But our systems are much more integrated with a lot more single points of failure. I doubt that bad updates were ever that unusual. But it wasn’t quite the same as it would have been in 1990 when there were dozens of different OS and virus software combinations and so on. One company doing one update would have only affected the few companies that had the wrong combination of systems that got a bad update. Now the combination of cloudflare and Windows is common enough that one bad update takes out thousands of computers in thousands of companies.

I’m mean, one way of looking at it is that the affected computers are now very well protected from viruses.

Considering how millions of computers are gonna have to be booted into safe mode and have OS/antivirus files tampered with, just wait until malicious actors start "helpfully" supplying USB thumb drive images that promise to deploy the fix automatically (alongside rootkits). Rootkits that might silently disable or bypass Crowd Strike entirely.

This is happening and CrowdStrike already has multiple page warning about various efforts.

Likely eCrime Actor Uses Filenames Capitalizing on July 19, 2024, Falcon Sensor Content Issues in Operation Targeting LATAM-Based CrowdStrike Customers

Falcon Sensor Content Issue from July 19, 2024, Likely Used to Target CrowdStrike Customers

From the second link:

CrowdStrike Intelligence has monitored for malicious activity leveraging the event as a lure theme and received reports that threat actors are conducting the following activity:

  • Sending phishing emails posing as CrowdStrike support to customers
  • Impersonating CrowdStrike staff in phone calls
  • Posing as independent researchers, claiming to have evidence the technical issue is linked to a cyberattack and offering remediation insights
  • Selling scripts purporting to automate recovery from the content update issue

No direct culture war implications, at least not directly left/right. However, this was easily predictable by readers of Michael Crichton or Ayn Rand, both names in the “up/down” culture war (to coin a phrase).

Crichton’s most famous work, Jurassic Park, was largely about chaos theory. When working with a complex system, that is to say one driven by logic and rules, an outlier can bring down a house of cards through emergent effects. John Hammond not paying for a team of programmers led to dinos eating people. Today’s a mundane version of that.

Rand had a lot to say about innovative producers versus free riders, and apropos to today, about smart people who can create or repair machines versus everyday people who can just use their interfaces until something goes wrong. When it does, the cynical cry of, “Who is John Galt?” escapes their lips.

In the classic book “Atlas Shrugged”, the phrase Who is John Galt is a cry of despair and hopelessness. It describes a situation wherein the pistons are removed from an engine making that whole metal mass of a car useless.

The pistons form a small part of a vehicle’s mass, but provide the entire reason for a (petrol) car’s existence. Similarly most great organisations and societies are moved by a small group of people — the innovators. When those are removed, the entire thing falls apart. And the engine is usually among the last parts of the car to give up. And when the engine gives up, usually you don’t find a replacement — you just sell the vehicle to scrap. When the small minority of truly creative, entrepreneurial, risk-taking people are removed from a society, the society completely falls apart.

John Galt is a symbol of that risk taking, entrepreneurial guy. And when he gives up, the despair sets in. “Who is John Galt” is a cry from the masses who are confused about what is happening and who are despairing to get back the people in charge [the people who can take charge of reality through reason and bend it to their will].

And the truly creative, entrepreneurial guy need not be a rich industrialist. He can be a worker. He can be an artist. He can be a banker. He can be a professor. It is not about their wealth, but about how much they move the status quo.

The American IT industry was hit hard by COVID. Businessmen, C-suite execs, saw their people remoting in from home and trying not to return to the office. These execs, many of them free riders, realized they could halve their costs by hiring remote MSPs from out of country for IT and relying on Crowdstrike to be their security bottom line. A flood of IT layoffs happened this past year, deflating IT wages and making entry level jobs scarce.

Then today, only people with the admin password or a modicum of critical thought could restore the most well-protected systems. Today, companies across the globe learned who their John Galts were, their Eddie Willers, their Dagny Taggarts.

Although, as to the left/right culture war, imagine if this or worse had happened on Election Day and all the votes had to be hand-counted.

The American IT industry was hit hard by COVID. Businessmen, C-suite execs, saw their people remoting in from home and trying not to return to the office. These execs, many of them free riders, realized they could halve their costs by hiring remote MSPs from out of country for IT and relying on Crowdstrike to be their security bottom line. A flood of IT layoffs happened this past year, deflating IT wages and making entry level jobs scarce.

I do think that it is slightly more complicated than that. First off all the lay offs of 80% of Twitter showed everyone that you don't need that many people to run a website. It was predicted by multiple of people that if Twitter didn't stop working other big tech companies would follow. Then there is the whole deal with Section 174 also that has affected the bottom line. Tech isn't unaffected by higher interest rates, when money was cheap they could amass people to be ready for "initiatives". Well not anymore.

I can give you the point of the free riders. The worst thing about them is that they actively make our tech worse to promote some number go up on their OKRs. Google is making search worse so people stay longer trying to find what they came to google for and watch more ads. Windows search always hits Bing when you do a search locally on your computer, just that it increments a number so a free rider can get a bonus. Just to take examples of search.

Your Section 174 link was fascinating. I feel that it underplayed the back story. It was sketched very briefly, but appears to go like this:

There are fiscal responsibility rules. If the US government passes a tax cut, the law should also include a tax increase in the future to balance the budget over the longer term. Legislators game this by writing a future tax increase that is stupid. Yes, it is in the law, but there is a nudge and wink that it will be repealed before it takes effect. This time the repeal never happened, so the deliberately stupid tax increase goes into effect.

This compounding disfunction bodes ill for the future of the US.

No direct culture war implications, at least not directly left/right.

Crowdstrike was the company who the DNC had analyze their network and blame Russia for Guccifer 2.0. I can write up a conspiracy theory about this being a result of the deep state panicking over the failed Trump assassination and forcing a patch to create a backdoor to cause a future major outage to maintain control pretty easily.

I don't think this is too apocalyptic, probably most computers will be fixed by Monday.

But you bet your ass that everyone lost a lot of money today and that it may take weeks (or months) for some businesses to get back to the black.

Does anyone disagree with me that the amount of value destroyed by this failed patch outweighs all of the economic value CrowdStrike has ever provided? Imagine working at a company that would have been better off never existing.

I disagree. Crowdstrike Falcon Sensor is meant to keep ransomware from happening, especially to (or through) the Internet of Things. Without it, at least some of the dozens of hospital systems which went down today would have already been hit by sophisticated unscrupulous organized criminals.

I feel sorriest for MGM, who got BSOD’d by Crowdstrike after getting ransomwared last year.

But you bet your ass that everyone lost a lot of money today and that it may take weeks (or months) for some businesses to get back to the black.

The market's reaction was surprisingly sanguine to this. CRWD stock opened 11% lower and stayed that way; almost everyone thought it would be down 30% or more. The Nasdaq was green for the first 2 hours and then went red, which could have been due to anything.

The economy is huge. Even when critical things fail, there is enough stuff that works, plus rapid response to fix the problem, that the damage is not as bad as the hype would suggest. Ironically ,a bigger problem entails a more rapid response to fix it, so it ends up being briefer or not as bad.

The cope is that this incident just shows how important CrowdStrike is.

Kinda like Boeing. They can have plane crashes, faulty parts, kill whistleblowers, etc... But we still have to buy Boeing planes – because we don't have a choice!

I'm less sanguine about Crowdstrike. Elon said he is ripping them out of all his companies. While the typical CEO drone probably won't do the same, Crowdstrike won't live this down. Maybe ever.

I predict a slow bleed out in their stock, although there's a good chance that internet morons bid it up higher over the next few weeks.

I think Elon did the right thing.

I have never heard about Crowdstrike. No computer I work with had it installed.

I totally understand that an average user is clueless and we need to protect him from his own actions. And yet, if this is such a necessity, why wouldn't Microsoft implement it directly in the OS?

Crowdstrike might be bleeding edge The need for bleeding edge is always overvalued.

It reminds me all times when everybody was trying to install antivirus software. Instead I always removed it because it only consumed resources and provided very little benefit. The best protection was to limit what user can do – do not install unauthorized software, don't even browse internet for fun, just use your work assigned software and web sites.

I think those who relied on third party antivirus software had worse outcomes because their users were more relaxed and less disciplined. At the same time those antivirus software makers got rich.

Probably the same has happened with Crowdstrike. Gradually Microsoft will implement something similar for no extra cost, everybody will realize that Crowdstrike is pointless. Until new challenges will come along and a new opportunistic company, playing on people's fears will convince to buy another scammy service.

And yet, if this is such a necessity, why wouldn't Microsoft implement it directly in the OS?

They did. The only thing missing from Windows integrated security is that it lacks the options to spy on users (breaking multiple privacy laws) and doesn't make it as easy to disrupt productive work by locking down the computer way too much. It also doesn't slow everything to crawl. Naturally corporate IT managers can't stand that.

Is the implication that the market would properly "punish" them for destroying more value than they've ever created? It could just as easily reward them for extracting rents for "malware defense" while making all of its clients worse off.

The price of any asset is the net present value of all expected future cash flows.

It's not about the stock market punishing a company. It's about the stock market trying to correctly evaluate how much other parties might try to punish the company. If we look at Boeing, we know that increased regulatory scrutiny is very unlikely to increase cash flows, and spectacular reputational damage is unlikely to increase future business. And so cash forecasts are updated accordingly.

sure, but my claim was CrowdStrike has probably caused more economic loss from this one patch than they have ever provided, which is somewhat orthogonal to a statement about their stock price

the fact that their stock price is not zero only indicates that the world's ability to hold them liable for these losses is minimized

(or that they can be held liable and that my estimate of the damage caused is way, way off)

Does anyone disagree with me that the amount of value destroyed by this failed patch outweighs all of the economic value CrowdStrike has ever provided?

I think it depends a lot on what your next alternative is. The morbid possibility is that CrowdStrike could be incompetent and also beat their customers rawdogging the internet. Even if this incident cost 1b USD, that's something like fifty major ransomware strikes. CrowdStrike could conceivably have blocked that many this year.

Of course, CrowdStrike isn't the only alternative. Businesses can use a variety of other protections and/or make themselves more robust to successful attacks. Whether they're more reliable or not is a !!fun!! question, but underneath that, there's a funner one: could businesses have made it? Contra a lot of reporting, I don't know that every regulated company has to use CrowdStrike specifically, but I do know that for even low levels of regulated industry it's a very common requirement that's accepted as a box checked, where alternatives that I could find required additional support not all IT teams would be able to provide.

There’s a reasonable case to be made that CrowdStrike isn’t a "real" company anyway: it’s a DeepState actor, worming its way into systems by enabling managers to check a box that satisfies regulatory compliance while giving wholesale control of their system to this opaque third-party.

I work in the industry and while I can confirm that regulatory compliance related to cybersecurity is theatrical bullshit, your assessment of CrowdStrike is completely wrong and nonsensical. It's certainly not the case for every vendor in the industry, but CrowdStrike's products and services do significantly reduce the risk of certain types of cybersecurity threats companies face.

This is effectively the argument that Lucas Critiqued.

In fact, it's almost exactly analogous to the Fort Knox example given in the article.

Why do you believe that CrowdStrike provides value?

Maybe it does but where is the proof? The half of the world didn't use CrowdStrike and how did they fare?

I would even say, let's do RCT to prove that CrowdStrike improves outcomes. It is perfect case when it could be done.

Maybe nobody wants to do such a test because they are afraid that it will show that CrowdStrike provides no value.

Remember masks during covid. The evidence is that they provided either minimal value or no value at all. And yet the government mandated their use in many countries. Sometimes people do stupid things on large scale.

I'm not saying CS provides no security, but it's hard to believe it provided as much global security as the damage it caused and that a competitor wouldn't have been better.

Sure. And asking how much security they provided requires addressing the counterfactual

So I keep trying to find any information on the technical aspects of this failure. As in, why is it bricking systems. I get that it's a driver that runs under the operating system, and it's failing to load. But why? I've only seen random reports that Crowdstrike literally pushed a corrupted file onto millions of systems, which is rather remarkable if true. If it was actually a bug, I'm deeply curious to hear what the bug was and how it slipped through.

To get really wild and speculative, lately it's been getting reported that Intel 13th and 14th gen I9 CPUs might be defective at incredibly high rates, upwards of 50%. These defects manifest in whole hosts of ways like BSOD, software crashes, and memory errors. I wonder if it's possible a defective Intel CPU borked the executable of an otherwise rigorously tested release. Like I said though, pure speculation. The nature of the Intel failures are still being investigated anyways.

This is not confirmed information, but I am hearing it on various technical grapevines and it seems plausible:

The primary bug is not new - the kernel-level driver that Crowdstrike runs (and has been running) has a dormant bug in the portion of it that parses config/data files. This update was "just" a config/data file, so deemed low-risk and put through fewer/simpler rounds of testing than a "real" update to their actual software. Whether it was a weird corner case or a malformed file, the kernel driver tripped over it and triggered the dormant bug. Since it's a kernel-level driver, crashing can affect the OS - and it did, generating an exception on a bad memory access (perfectly routine type of bug, but with privileges!) so the OS crashed.

Lol that is amazing. Sounds like the most plausible explanation, but maybe even worse because it seems like that should have been caught in a dev or staging environment

Forget about dev or staging, there's no excuse for not fuzz testing your config parser in current year plus nine.

Im not in a position to confirm but that seem quite plausible and dove-tails with some of what I've heard.

For some reason i find myself thinking of this old XKCD 😉

Ok so there's an update on what happened.

The exact crash is caused by dereferencing a null pointer the offending assembly is readable by anyone, and it is as follows mov r9d.dword ptr [r8], the key is that the value of r8 is 0000 0000 0000 009c 9c is an offset of some sort set earlier, so it's derefrencing a null pointer. The pointer is NULL because the value in the file C-00000291.sys was published to be all 0s causing r9d to get loaded as all 0s

So the offending assembly probably looks like

read r8 C-00000291.sys (some offset)

add r8 9c

mov r9d.dword ptr [r8]

causing the bug.

From this, it kind of sounds like rather than having an on-disk data representation that would be parsed and converted to an in-memory data structure, they just loaded the file and accessed the raw bytes as a data structure with internal pointers. Which is... an approach, I guess.

It's an executable; that's how executables work.

Eh, not really? Executable files have structure in them other than raw code and still have to be parsed by a loader. A file that's all zeros should fail to load. (Yes, I know DOS had .com files with were just code blobs loaded at a fixed address and immediately executed and I'm sure there are even more ancient examples of that sort of thing, but surely Windows kernel modules can't work like that.)

Anyway, the rumors I've read said that it was actually a data file and that's why they considered it acceptable to deploy it on a Friday -- the assumption being that changing configuration without rolling out a new version of the executable wouldn't break things too badly.

That might be how executables in an operating system work. Wouldn't be how extremely low level BIOS or ROM code that is meant to be executed before the OS loads would work. I can't say for certain exactly how that works these days, but when I was troubleshooting some BIOS code on an old computer of mine, I found myself decompiling a VGA BIOS. And that basically works by being in a certain memory block, it begins with a consistent signature to signal "Yup, there is code here" to the motherboard BIOS, and then it begins loading and executing instructions at a certain offset to initialize the card. Fun fact, you can actually reinitialize the VGA BIOS with a short assembly program that just CALL's to that location if memory serves.

What you are describing sounds more like a boot sector, i.e., raw machine code meant to be read from bootable media and executed directly by firmware (the mobo BIOS in your example)

I’d be surprised if in any modern operating system, executables (even those loaded and run at boot time) were handled that way. Then again, one is reminded of the old chestnut about idiot-proofing software…

The problem with turing machines is that pretty much everything becomes equivalent at high enough levels of generality. Windows EXEs (and DLLs) have a specific format that make it impossible to load an empty or (most) malformed files, but if the surrounding format is correct enough you can absolutely have it followed by a bunch of nonsensical instructions and memory locations -- there is a checksum, but (infamously), it isn't actually mandatory to load or run.

Worse, there's no rule that your executable is the only place that such instructions can come from, and few architectures try. Even in Harvard architectures like Atmels or PICs, there are specific instructions to transfer from the data bus into the program and vice versa. Modern operating systems on von Neumann architectures try to stop you from doing so by accident, by setting memory pages as either instruction or data, and in modern Windows machines further isolating data instructions with DEP, but it's ultimately just a set of flags.

There are arguments against doing this, in favor of having a having your base program load from more conventional configuration files with a strict format (eg JSON), or even having a very limited programming language that your core driver then 'runs'. They have some tradeoffs! But ultimately the problem is a lot more boring: in each case, you have to be able to recognize and respond to a corrupt file. And that's a solved problem! But you have to recognize it.

More comments

I'm pretty sure I could write a C program right now that would run in Windows 10 that will load and run arbitrary assembly instructions from a binary file. The C program might have all the trappings of a proper Win10 executable, but the file it loads and runs sight unseen wouldn't. I'm pretty sure that's what the Crowdstrike driver is doing with the file full of 00's.

Not really. The linker and a bunch of other transformations are going to happen before any of your instructions run. Dumping and loading bytes of a structure straight out of memory has long been considered a lazy and dangerous thing to do; no one is surprised that this sort of bug arose from it.

Apparently, the corrupted file was just filled with nulls:

https://twitter.com/jeremyphoward/status/1814364640127922499

I'm trying to image what might cause that; truncating the file and then failing to write it? My filesystem-fu isn't really up to par.

Wasn't there an old joke about an MBA cutting costs in half by getting rid of the 1s and standardizing on 0s?

Saving a file using a filesystem that journals metadata followed by computer crash that happens before the file contents are flushed is one way to achieve it.

In tech, staying at a job for more than 3 years is seen as coasting. Devs are increasingly expected to do everything, because 'everyone should be full stack' and everything that isn't feature development (testing, staging, canaries) get deprioritized. Overworked novices means carelessness, carelessness creates mistakes.

This may be true amongst the Dev Set, but it's very much not once you get outside of that small corner of 'tech' and into the infrastructure side of things. While there are plenty of greenhorns puttering around, there are also the true gray beards who have been in the same position for decades and know literally everything about the systems they administer (and design and build).

I'm a network engineer and there are enough grizzled old men on my team that our collective experience no doubt stretches into the centuries, and several of these guys have gotten a big chunk of it at this one place. We just had a guy retire last year who started in the late 70s...

That bit jumped out at me as well. Even amongst the Dev Set that strikes me as a very FAANG / SV centric point of view.

If what is being is reported is true and they released some unrunnable or improperly formatted file, I can’t even comprehend that level of incompetence. There is a lot of bullshit at my company which is also dealing with many of the issues you’ve addressed in your post, and of course we have incidents, but something so basic being released with such insane permissions would not be possible at my workplace. Of course that’s discounting any malicious actor, but the number of QA cycles and slow rollout that we go through would have caught something like this 5 weeks before it sniffed release.

Something or someone is deeply rotten at crowdstrike. They need to make a big-time firing or I predict that people will start fleeing in droves.

This seems to me like a fairly usual level of competence from a bolt-on-security-as-a-product or compliance-as-a-service company. Examples:

  • CVE-2016-2208: buffer overflow in Symantec Antivirus "This is a remote code execution vulnerability. Because Symantec use a filter driver to intercept all system I/O, just emailing a file to a victim or sending them a link is enough to exploit it. [...] On Windows, this results in kernel memory corruption, as the scan engine is loaded into the kernel (wtf!!!), making this a remote ring0 memory corruption vulnerability - this is about as bad as it can possibly get". Basically "send an email with an attachment to pwn someone's computer. They don’t have to open the attachment, as long as they have Norton Antivirus (or anything that uses the Symantec Antivirus Engine) installed".
  • CVE-2020-12271: "A SQL injection issue was found in SFOS 17.0, 17.1, 17.5, and 18.0 before 2020-04-25 on Sophos XG Firewall devices, as exploited in the wild in April 2020. [...] A successful attack may have caused remote code execution that exfiltrated usernames and hashed passwords for the local device admin(s), portal admins, and user accounts used for remote access"
  • Okta data breach a couple months back: "For several weeks beginning in late September 2023, intruders had access to [Okta's] customer support case management system. That access allowed the hackers to steal authentication tokens from some Okta customers, which the attackers could then use to make changes to customer accounts, such as adding or modifying authorized users."

It's not that it's amateur hour specifically at CrowdStrike. It's the whole industry.

I have always been of the opinion that antivirus is a poor idea, and at best, a half-baked solution preventing you from adopting better solutions, such as sandboxing/virtualization and general human security hygiene. I haven't run an antivirus (besides Windows's built-in Defender) in years on any of my computers or phones, and I've never gotten malware on my systems simply because I don't open any sketchy apps or files, and if I do, it's in a virtual machine isolated from the rest of my system.

That an entire industry (the antivirus industry) exists based on the premise of a bad idea that is not only ineffective but adds massive attack surface simply because attackers can exploit what is essentially a privileged system component with deep access to all parts of the system - a cure worse than the disease - should be a lesson in how easy it is for someone to get the basics of a skill (such as security) wrong.

The problem is that simply receiving a text may count as "opening a sketchy file". You really can't expect every boomer pecking at a computer to know the ins and outs of security.

This is not to defend this particular software, but your view leaves out some things as well.

Bad example? If you're targeted with zero-days like Pegasus, an antivirus software is not going to stop it. In fact the standard defense for this sort of thing is what I've advocated - isolation of system components via sandboxing/virtualization. I'm not sure what your argument is.

AV can at least detect anomalous network traffic or unexpected processes, which is obviously not as good as preventing the infection in the first place but is still valuable.

In this case, the systems were sandboxed - FORCEDENTRY escaped the sandbox. Sandboxing isn't a magical technology without vulnerabilities.

Would antivirus have actually detected this infection? Ignoring the fact that phones don't usually run antivirus (because they employ sandboxing security measures), in the case of FORCEDENTRY, the exploit was discovered because Citizen Lab specifically examined the phone of an anonymous Saudi activist. They don't say what exactly led to the phone being examined by them, but I'm willing to bet that it exhibited signs of infection that any general-purpose antivirus like McAfee wouldn't have detected.

Yes, sandboxing technology can still be vulnerable, but antiviruses are not a better security practice than sandboxing. Moreover - since you brought up a targeted spyware attack - if you're being specifically targeted by nation-state actors aided by NSO Group, you need to up your security anyways. So your comment that

You really can't expect every boomer pecking at a computer to know the ins and outs of security.

immediately after discussion of FORCEDENTRY confused me, because if your threat model includes zero-day attacks like FORCEDENTRY (for example, you're a political activist, journalist, or whistleblower), then yes, I do expect such a person to know the ins and outs of security. They should stay on top of their game, because their life literally depends on it. At that level of threat modeling, if you're genuinely worried about attacks from well-funded nation-states, then security is not something you can just ignore and expect to have taken care of for you.

Yes, sandboxing technology can still be vulnerable, but antiviruses are not a better security practice than sandboxing.

It's not one or the other.

Moreover - since you brought up a targeted spyware attack - if you're being specifically targeted by nation-state actors aided by NSO Group, you need to up your security anyways.

Bringing this up as an example was my mistake since it seems to have derailed the conversation.

There are plenty of vulnerabilities out there that are not zero days. There are plenty of systems out there that are vulnerable to such attacks. Not everything is patched as soon as the CVE is published and not every system is updated as soon as the patch is published. It's a simple fact of life that there is a time period between a vulnerability being disclosed and all systems being updated, even if those systems are enrolled in some kind of regular update scheme. Arguing against the need for at least detection and monitoring for threats because you have a lot of faith in sandboxing does not make sense.

More comments

A general rule: the further a software product is away from "engineering candy", the worse it is.

Software engineers are some of the most entitled, overpaid people on the planet. (I should know!) They have lots of career options.

To get good engineers you need to either pay an outrageous salary or have an interesting product like a video game. Want to find engineers to work on your compliance software? Good luck. Hell, even Google engineers making 400k/year can't be bothered to work on essential but boring products, preferring instead to chase shiny baubles.

No one wants to do the dirty work where good job means not messing up.

I think the problem is that "good job" doesn't mean "not messing up" in the context of these compliance-as-a-service or security-blanket-as-a-service companies. Instead, "good job" is "implement as many features as possible to a level where it's not literally fraud to claim your product has thay feature, and then have a longer checklist of supported features in your product than the competition has so the MBA types choose your product".

CrowdStrike's stock price is only down by about 10% today on one of the highest-impact and highest-profile incidents of this type I've seen. I'm pretty sure their culture of "ship it even if it's janky and broken" has netted them more than a 10% increase in net revenue, so it's probably net positive to have that kind of culture.

Their net revenue is under a billion a year. The total economic damage caused by this single bug is almost certainly larger than the total net income of the entire history of the company. In fact, it is almost certainly larger than the total gross income of the entire history of the company. I do not know where the valuation is coming from, but it certainly isn't from their revenue figures.

Lol P/E of 644.

But it's a hyper-growth company bro, surely they'll be able to pivot to making money once they've captured the full market bro.

Yeah but if they're not liable what relevance does that have to their share price?

I don't know if they're liable or not. I doubt Crowdstrike knows if they're liable or not.

the further a software product is away from "engineering candy", the worse it is.

To get good engineers you need to either pay an outrageous salary or have an interesting product like a video game.

I mean, you could get good engineers with a video game project, but for that you have to be willing to also pay them the outrageous salary. Video game projects are more art than engineering, requiring more designers than engineers. And the brilliant engineers won't work for that much below market rate; if that were their goal they'd go into research or try to get into an early-stage startup, not join a project that's just the application of an existing engine to a new gameplay design. The game projects that appeal to engineers don't sell enough for AAA development, they're nerd games like Factorio or RimWorld (sorry friends).

Not that game companies don't capitalize on the appeal of their projects to talent. They just capitalize by taking lower-tier but motivated engineers/artists/designers and running them into the ground.

I incidentally just learned about the Okta breach yesterday simply by getting frustrated with it and searching on Twitter evidence on whether everyone else hates using it continuously as much as I do.

I have the opinion that the more data you give out, the more likely it will just get breached. Especially personal data meant to authenticate your identity. The best thing to do would be to not give data out at all - data that doesn't exist, can't be stolen - but most of the rest of the world doesn't think the same way, and are extremely unlikely to question why we have normalized people giving away their data without a second thought.

Don't they deploy updates like this in a development evironment first to test for exactly this kind of thing? I work in very low-level, mostly unimportant IT and I sweat breaking a single website that gets 100 visitors per month. How does something as big as this not get tested first?

They don't stage releases sending them out to limited groups one at a time? They do one global update and hope for the best?

There's such obvious ways to limit the impact of this sort of screwup.

I'm going to play Karnak the Magnificent here and say they do indeed do staged rollouts.

They just don't properly check if one stage has succeeded before moving on to the next.

Rumors suggest that it may have been rolled out Friday morning local time.

Of course, a slow rollout is pointless if you have no canary process and no means of determining if you just bricked all Australia...

I thought this is exactly why they rollout updates instead of distributing them all at once. Do we know for sure there was a rollout or could they have mistakenly pushed this everywhere at once?

In this case I think it depends on what is being pushed. You have to keep in mind that this is a security tool specifically promising and designed to implement rapid defense against zero-day security exploits. Holding off for a week or so on a threat under active exploitation is not what they are being paid for.

Yeah, the paranoid option is that there was some serious zero-day that they were trying to react against, it worked fine on the development environment, and they made a tradeoff of the risk of this sort of incident against not pushing the big red button.

But being derpy is always an option.

That’s a good point, but they have to have some kind of staging environment or slow rollout right? You can’t just release to all customers at once, that’s absolutely insane and asking for something like this to happen even if it’s security-critical.

It's mid-July. Likely an intern bypassing a safety check to try to get his project completed on time.

Note that it's crowdstrike, not cloudstrike. Doesn't detract from the post that much but just thought it was worth pointing out.

ClownStrike, I think

Thanks, fixed

Can you remove the strikethroughs, at least after the first one? It's a bit jarring.

It's already over. The biggest non-crisis ever even if it was the among the most widespread.

No direct culture war implications, but goes to show just how much of a house-of-cards the tech ecosystem is. 1 little, simple, stupid bug can bring the whole world to a halt. Yet, the industry continues quarterly-earnings chasing.

That seems overdramatic. I have not noticed any disruption at all; if not for all the headlines this morning I would have never known about this. An upgrade was rolled out and it was fixed. This requires manual intervention of servers, which is why IT exists as a profession in the first place. It's not a crisis like on the order of Covid or 2008, but more like a mass disruption. I think too many people are overreading into this as some sort of harbinger of the awaited collapse, and really it's not.

Airlines were grounded; again, this is a common occurrence. There was a similar incidence as recently as 2023 when many flights were grounded https://www.reuters.com/world/us/why-us-flights-were-grounded-by-faa-system-outage-2023-01-11/

it's bad, no doubt, but the mass-grounding of flights is something that typically happens every 2-3 years.

End of the day, tech workers are treated as disposable labor. Executive bean counters are divorced from the product. And the stock price is the only incentive that matters.

The fact they are paid so well and exhaustively vetted in the hiring process suggests they are not disposable. Companies invest a lot of resources in new hires . There is also a loss of perspective in that people forget the other 3650 days of the past decade in which there is no major failure, but a single failure is suddenly a major indictment on the entire tech industry, as opposed to something more mundane like a mistake.

Crowdstike stock was only down 11% today, which is far less than expected given that it has been implicated in the greatest IT failure ever. By comparison, Meta stock fell 15% in a day last after it missed the highest of earnings estimates. This is reason to believe it's not as bad as the overly dramatic language would suggest.

One would hope such companies learn from past mistakes, but as tech changes, consequently so do the mistakes. So I can expect incidents like this in the future.

I had a flight canceled today. I am fucking livid. This was over 12 hours after the rollout. Luckily I was able to get rescheduled onto a flight tomorrow, but frankly I have no confidence that that flight will happen either.

I was just going on a silly vacation. I cannot imagine how I would feel if I missed something important. There will never be justice for this. In a fair world, Crowdstrike would be sued into bankruptcy like Purdue Pharma. I'll be lucky if I get a drink voucher out of this.

Are you sure you can get nothing? Last time an airline messed up my connection I got around 800 euros out of it

in the United States, airlines aren't legally required to compensate customers for delays at all. I had a United flight recently delayed by eight hours and received a $15 lunch voucher and $100 in airline credit though.

I can't help but think about this post that was linked on the SSC reddit a few days ago: https://matt.sh/panic-at-the-job-market

(long, rambly post that I don't fully agree with but it did say a few interesting things) In particular these two quotes:

Modern tech hiring, due to industry-wide persistent fear mongering about not hiring “secretly incompetent people,” has become a game divorced from meaningfully judging individual experience and impact

...

Such job descriptions also means: your job is physically impossible. You will always feel drained and incompetent because you can’t actually do everything everyday. You will always be behind because each of those bullet points can be multiple days of work per week just on their own (plus, how are you supposed to be productive in 35 different areas requiring months to years of experience if you actually want to be good at each task?). So, from day 1, you will already be about 4 months behind on your expected job responsibilities and you’ll never catch up. It turns into an endless game of managers and executives saying you are “underperforming” because you have 18 primary tasks, each primary task requires 4 to 20 hours of effort, and every manager wants their task done within 4 hours. You are setup to fail. What’s the point?

Maybe a point is some companies just shouldn’t exist if they can’t afford the fully staffed professional teams required to build and maintain their products? The worst secret in tech is amateur developers are happy to act like entry level workers across 20 arbitrary roles for years (in the absence of never having enough time to focus on building up long-term experience or best practices). You can’t get gud if you are always rushed from task to task without any chance of leveling up knowledge and capability through “deep work” as we would historically expect of professionals.

I don't think it's like that at every company, or even the majority. But there are certainly some companies like that. They, in theory, care greatly about their tech workers, because the salaries are high and they have a vague understanding that tech is important. But they don't have a good system for actually hiring good tech workers. And then, once hired, they use them all as generalists, moving quickly from one thing to another, with no chance to actually develop expertise or fix deep underlying issues. And they are never given any kind of decision-making authority in the company, only responsibility to "just fix whatever breaks."

I think that behavior happens the most in companies that are not "tech companies," but still use tech. Banks, airlines, large retailers, that sort of thing. They need tech to function, but it's just a cost center to them- they want to just pay a fixed price per month to "handle tech" and then not think about it ever again. And it seems like those are the ones being bitten in the ass by this thing, because it turns out that running a windows server with third-party antivirus on it with automatic updates is not actually very secure! I wonder if we'll see any restructuring, or if this sort of thing is just going to happen every so often forever, as companies get blindsided by tech issues that they don't understand and never cared to try and understand?

i think the culture of secrecy is the bigger problem. they are paid lot and expected to not blabber to the media if they expect to be employed now or in the future by other companies

Wikipedia reports that 5.9% flights were cancelled worldwide. It's definitely a lot of flights but also not that much on global perspective.

Twitter had flightradar24 animations showing flights disappearing with Community Notes saying that this animation is fake and not from CrowdStrike fault event. You wouldn't really notice 6% decrease visually or would notice only a slight reduction.

People love to lie on twitter for dramatic effect.

If we assume 6% reduction of global economic activity for one day, it certainly is loss of billions of dollars. And yet it is less than one extra holiday per year.

I was going to do a lot of stuff at work today. Was.

Interestingly this crash has seemingly barely affected Finnish businesses and organizations at all (apart from cases where they have projects with foreign companies, of course). Apparently there were some minor glitches at the system of the bank I use, but I didn't notice it at all.

I wonder if it's simply that Finnish companies are patriotically committed to using F-Secure/WithSecure solutions above all others...

I’ve never heard of CrowdStrike and I’ve worked as a programmer for 25 years, so I assume more or less nobody uses CrowdStrike in Finland.

Heard much the same from a programmer friend.

Finnish = Linus Torvalds = Linux servers ?

Possibly a part of the explanation, too.

Yep, you're right.... i was going through twitter and got duped.

I don't even understand why something like clownstrike is necessary in 2024. It should be possible for the OS to be locked down to the point where it's not necessary to have an anti-virus running. And if you need some other security system because you are worried about zero day exploit from nation state threats then you should really consider your threat model because the clownstrike system is effectively a malware distribution platform. I guess its fine if you trust clownstrike and the US government but its a far from ideal situation. Clownstrike seems to have a very nice relationship with the US security state. For example they were brought in to do the hacking investigation by the DNC and provided attribution to Russia.

OS vendors should really expose some kind of interface that allows security vendors to perform these deep inspections 'safely'. I think linux has EBPF which I think some vendors have been using for providing file system monitoring and network monitoring.

Also, the SOC2/etc compliance mandates a lot of this stuff. We run most of our software on Fargate ECS where the compute is completely managed by AWS. I've been using this as an excuse as to why we can't run file monitoring and other garbage on our systems that use Fargate. I also suspect why these managed docker/managed kurbernetes systems are popular because potentially you can avoid some of the tickbox security work. We also run all of our containers with a read-only rootfilesystem so I don't even understand the threats that a file system monitoring system would be trying to remediate in our situation. Technically some kernel exploit could allow the root filesystem to be modified even if its read only or AWS employees could fuck with us but I suspect in these cases the file system monitoring could also be trivially bypassed.

I don't even understand why something like clownstrike is necessary in 2024.

Clownstrike and all the other security stuff is the triumph of the security engineers and MBA types over users and cowboy developer types. Security incidents happen. Security engineers blame users and cowboy developer types, come up with software to make computers crappier and less useful. MBAs (especially MBAs at companies making this malware) call this "best practices" and push to have them required by corporations and governments. Developers and users complain that their computers are slow and don't work, the MBAs and security engineers say 'that's how you know it's working'. Then something like this happens and the cowboy types indulge in schadenfreude.

I don't even understand why something like clownstrike is necessary in 2024.

Because, just like with DEI and other stupid corpo bullshit, business necessity has nothing to do with efficacy. You do the rituals and check the boxes because someone somewhere figures this lets the company cover its ass. Whether there was an actual threat of ass exposure to begin with doesn't even get considered.

Checking to see if my flight will be delayed due to this. It still says "on schedule", but following the chain of "where is this plane coming from" backwards in time to see where my plane is, I see one flight where the expected departure time is before the expected arrival time of the airplane.

A website that follows and shows you the chain of previous flights of your plane sounds like a pretty cool idea

I do hope the fallout from this crap will be immense. Cloud was bad idea from beginning. This type of cloud security too.

This is the opposite of the cloud.

It is Software-as-a-service, but the processing wasn’t being done on someone else’s computer.

This isn't "cloud" in any meaningful sense.

Indeed, if these computers were in the cloud, they'd be fixed much faster.

Well, not cloud, but internet in general.

These machines all updated something, because they are connected to the internet and set up for automatic updates.

People learn pretty quickly that automatic updates are a terrible idea. Even if the update doesn't screw up your data or your workflow, e.g. by taking away some feature you were depending or crapping up the UI, it's likely the update process will kick in at an inconvenient time (like in the middle of a presentation). So they turned them off. Security people started crying about unpatched bugs, and got enough corporate power to get automatic updates considered a "best practice" (when it's not), and here we are.

Automatic Windows updates destroyed two of my work laptops at my last job.

I've had Windows 10 updates fuck up some of the older software I have running for my job.

And people wonder why I turn Windows 10 updates off.

Now I'm going to have to fight off a Windows 11 upgrade, so as to not fuck up said software. You'd think local IT would be more paranoid about just gleefully installing whatever it is Microsoft tells them too, but...

I can't speak for your IT department, but in the past we would always test updates across a cross section of the business before rolling them out to everyone. Maybe like 10% of the computers would get the test updates, and we would only deploy if we had no issues on the test PCs. That's really all you can do though, sometimes issues come up even with testing.

Automatic updates are the worst thing . Everyone hates them yet companies do it.

The problem is that no automatic updates is also a terrible idea, as a majority of systems don't get patched, ever. The ideal is manual updates but responsible companies/admins testing before deployment, and sadly I don't think that's gonna happen. The second best is gradual/tiered deployments with the ability to opt out, which is more realistic but still require more effort than many companies are willing to provide.

I personally think that "no automatic updates" is better than the current hellscape of "lol we can break your device at any time", even with the problems it causes. I'd rather have hella security issues on the Internet than have my stuff randomly break (or just get worse) without my intervention.

"Internet was a bad idea from the beginning" is certainly an interesting argument.

I can definitely agree that canary-less fast global rollouts were a bad idea from the very beginning though.

How long do you wager it'll be before a major car company [thinking of Tesla here but I'm pretty sure they all do this now] bricks a significant number of its electric cars by pushing a bad update (rendering the car unable to start)?

That seems best case. What if it bricks while driving?

Probably highly unlikely. I have worked on mission critical software. While it wasn't automotive it was in a similar field. The code I wrote took six months to reach production. At that company we wrote maybe 5% as many lines of code per work week compared to a normal company. There was also extensive testing.

There may be individual events that happen. Mass brickings are unlikely.

Considering the overall quality of automotive software is 100% garbage I'm not as certain a massive screw-up would be as unlikely.

More like, for all of its benefits the internet has always been, and will always be, a point of vulnerability.

Hmm centrally managed, by a third party, not on premises, critical security infrastructure with kernel access? There is definitely reading of cloud service that describes it.

The machines are on prem. That's the whole point.

If the machines were off prem they would be managed by some company with at least basic sysadmin competence and it would merely be a major annoyance to fix this. As it is, every mom and pop with a moron for an IT department is going to have to fix it themselves.

My company’s shiny new ERP system is hosted by our vendor, a large and growing company which sells to many industry verticals. The system is still down.

If our little SMB IT department had been running it on premises, we would never have installed endpoint protection on our servers. We may have had all kinds of other problems we couldn’t hedge against because of our scale, but we have the good sense to weigh the risks ourselves instead of complying with a customer’s backside-covering audit checklist.

You're right that I failed to consider that there are tiny cloud shops out there. When I was talking about cloud I was thinking about the big three.

This is like seeing a jet plane crash in the 1960s and being like "this idea will not work" or the Titanic sinking and thinking the same thing. Enough companies rely on such services that evidently it's worthwhile despite these risks.

Enough companies also relied on massive amounts of lead in the fuel they use for decades. In the digital era - not having all your data and services under your roof will forever be a bad idea. It's just that the beancounters were tired of paying those pesky sysadmins a livable wage.

I am not against the concept of services per-se. But the critical ones should always be self hosted. And moronic, useless antivirus part of the security theater is up there with critical. Anti virus hasn't been needed on windows for a long long time.

The best comment on all things cloud is way before the cloud was even a concept:

With sufficient thrust, pigs fly just fine. However, this is not necessarily a good idea. It is hard to be sure where they are going to land, and it could be dangerous sitting under them as they fly overhead

Once again, rawdogging the Internet pays off!

If everyone uses this, what determined who got hit? Did they do a random staged rollout and stop once the problems started?

Pure hearsay--but my IT guy says "if your system had Crowdstrike installed, and it was on and running automatic updates when the updates was pushed, then you got hit. If your system happened to be off, power-cycling, delaying updates, etc., then you missed it, and the actual fix was rolled out very quickly to prevent further problems."

So now "zero day" protection is also a zero day exploit.

Something something security monoculture? Truly critical infrastructure should probably be running multiple operating systems on vendor-diverse hardware in parallel, I guess?

If your system was up when the update rolled out in the afternoon, and you turned off or reset your computer before the rollback patch, you got a BSOD easily fixable by anyone with the admin privilege.

Part of security is a monopoly on force — sorry, on access — so nobody dumb can infect the system, and few people had the privilege. I was one of the clever few who could boot with a Windows installation USB, delete the affected files, and be back up in minutes. Whereupon I was asked to get other PCs up in our building, which I gladly did.

On reddit, someone said they’d been speaking with their crowdstrike security rep the previous week, who said they had a beta for the new version which was getting BSOD on some windows systems, so they weren’t going to push it out until the bug was squashed. It’s assumed in IT the bad update accidentally got into global distrib.

Who is John Galt?

Who is John Galt?

People old enough to have done things like "boot from a USB drive," but not so old as to be confused by computing devices generally?

Thirty years ago, relatively few undergraduates brought their own computers to college, though most had access to some kind of computer "lab." Twenty years ago, most undergrads brought their own computers to college. Ten years ago, it was common for many programs of higher learning to "give" students a laptop for curricular use, testing, etc. Today, I get a surprising number of students whose only computing device is their cell phone, or a similarly hobbled tablet-style appliance. They live in walled gardens and think that computing begins and ends with "apps." Throwaway consumption devices are, slowly but surely, crowding from our collective consciousness the general purpose (and modular!) machines that delivered the Information Age.

And in some ways, I suppose, that was always the goal ("it was always the plan to put the world in your hands...")--just as we don't need everyone to change their own oil, or know how to fly airplanes, we don't need everyone to be using desktop computers. But in much the way that the average American utterly fails to understand or, therefore, appreciate the systems that keep them fed, keep the power on, etc., I suspect that failure to even slightly understand the technology on which our civilization functions contributes to some pretty distorted perspectives--on the world, on life, on politics, etc.

I was one of the clever few who could boot with a Windows installation USB, delete the affected files, and be back up in minutes. Whereupon I was asked to get other PCs up in our building, which I gladly did.

Sounds like your IT security is subpar. No drive encryption and USB boot devices not blocked? This means anyone can exfiltrate the contents of any of the drives.

Nonono, they are clearly following Best Practices (tm) -- after, they have Crowdstrike!

security monoculture?

Tangential, but it's shocking how much small differences can impact results. In my industry, people decorrelate WTI from Brent, and then Brent from other Brent, by using4% instead of 5% stoplosses. They then make the full range 1,2,3....% on each, then bottle them up into different ensembles, and after a few days they show massive divergence.

Anatomy of a Deboonking: Why Debunking Stories So Often Fail to Persuade

So I'm looking at the Motortrend website looking for reviews as I'm shopping for a car for my wife. And I see this article purporting to independently test a "rigged" matchup used in Tesla marketing. Tesla posted this video claiming to show a Cybertruck Beast, towing a Porsche 911, defeating a Porsche 911 in a drag race. Right off the headline, where they state it isn't "rigged" this time, the implication is that Tesla (and that rascal Elon) lied to you! But upon actually reading the article, I'm left kinda cold. It seems to me like their effort to debunk the race just proved to me that the race was plausible!

Right off the bat, Motortrend admits:

At just 2.5 seconds to 60 mph in MotorTrend testing, Tesla’s three-motor electric truck beats every Corvette, every McLaren, and all but one Lamborghini that we’ve ever tested in the industry-benchmark acceleration test. That’s without a trailer, of course.

That's crazy numbers. It's insane to build a pickup truck, or even some kind of weird SUV thingamajig since it's only kind-of a truck, that can do Corvette and McLaren numbers on a drag strip! That's a category destroyer! It also tells us that, unloaded, the four passenger plus mucho-cargo Cybertruck will stomp on a two passenger and a duffel bag 911 in most cases. That's a big advantage. But ok, that isn't winning while towing a 911, so let's test it. Motortrend says that...

We approached our drag race with a scientific sense of curiosity. Regardless of which vehicle won, we wanted to understand how it nabbed the victory. We also gave the Cybertruck every possible advantage—within reason—since we assume Tesla also did the same.

At core, this meant that they picked a lightweight trailer, a lightweight 911 model to tow and removed all reasonable weight from the 911 being towed, the slowest 911 model to race against. Resulting in...

All in, our race pitted 11,268 pounds worth of stainless steel, aluminum, lithium, and bristling muscle propelled by 845 horsepower against 3,488 pounds of aluminum, carbon-spewing flat-six, and dad bod powered by 379 horsepower. The only thing left to do was race.

And the results come in and are instantly treated as conclusive!

Here’s What Tesla Didn’t Show You We ran six quarter-mile drag races, and each one had the same outcome: The Porsche 911 Carrera T wins and the Tesla Cybertruck Beast loses. In the world of drag racing, it’s not a particularly close race, either.

Notice the implication that Tesla lied, "Here's what they didn't show you." The implication of deception is used throughout, that this video amounted to a deceitful lie, that Tesla was taking advantage of you, the gullible potential customer. And the conclusiveness: it wasn't particularly close, by drag racing conventions.

We then go into a series of potential alterations to the format. Does it matter if we put the Tesla's best run against the 911s worst? No. Actually, Tesla only ran an 1/8th mile rather than a quarter mile, does that change the result? No, it doesn't. There is just no honest way to get the Cybertruck in front for more than about ten feet. And at the end of the article, Motortrend, that pinnacle of journalistic excellence, comes down hard editorializing against Tesla:

No matter how you slice it, present it, or asterisk the claim, Tesla and Elon Musk’s big brag doesn’t hold up. There’s no scenario where the Tesla Cybertruck Beast pulls a fully functional Porsche 911 Carrera T across the quarter mile before the Porsche 911 Carrera T gets to the finish line. Even if Tesla had originally disclosed it only staged an eighth-mile race, the video seen by customers, fans, and curious observers is highly misleading. ...it was reckless for a corporation worth billions of dollars to make an unsubstantiated claim to millions of people...Here’s the truth: A Tesla Cybertruck cannot tow a Porsche 911 Carrera T over a quarter mile quicker than the 911 Carrera T alone can run the race. Add it to the long list of broken Tesla promises. [Emphasis Added]

The moralizing tone and catastrophizing language is kinda overwrought here, n'est pas? Tesla may have theoretically deceived "millions" of viewers, but they only shipped 12,000 Cybertrucks in 2024 as of July, while Porsche only sells around 1,000 911s a month. The people whose buying choices may be actually impacted by the comparison are negligible. And, at any rate, someone who can afford a $125k 911 or a $100k Cybertruck is (hopefully) likely to be a sophisticated consumer who will do more research than just watching a Youtube video about an irrelevant occurrence and making their decision based on that advertising video. No one is being seriously impacted by that rascal Musk's awful DECEIT and FAKE NEWS. So why treat it that way?

And in any case, I came away from the article, despite Motortrend's laudable efforts at technical rigor, thinking that Tesla's claim was more true than it was false anyway! It was maybe not literally true under laboratory conditions, but those very conditions explain why it would likely be true in day to day life for the average fuckboi who blows six figures on a Cybertruck. Right at the beginning of the article they admit:

From the moment the words came out of Musk’s mouth, we were skeptical of Tesla’s towing/drag race video. Extraordinary claims require extraordinary evidence, and while Tesla did show the full, unedited race from four angles, that wouldn’t stop the Porsche 911 driver—presumably a Tesla employee—from ensuring the Cybertruck won. If the race was as close as Tesla’s video suggested, a soft launch, a slow shift, or a slight lift of the accelerator in the Porsche could be the difference between a Tesla loss and a victory.

Motortrend, of course, uses their own drivers. Who are presumably pretty fuckin' good at driving, pretty well educated in drag strip technique, and doing their damndest. Later in the article they note, when reviewing technical reasons for the result:

The manual-transmission Carrera T has a 3,500-rpm limiter at standstill, and on a sticky, prepped drag strip, launching quickly requires getting off the line without letting the revs fall. Drop the clutch too fast, and the engine will bog, falling out of its powerband. It takes a slow, carefully modulated clutch release to get the perfect launch, which keeps the engine on boil and extracts a small amount of slip from the tires.

The vast majority of Mottizens, who by and large are smart and technically educated people who (should, by rights) eventually be in the market for something like this, might understand all those words, but haven't engaged in the practical activity of doing that activity. I've driven some bitchin' cars, but I've never clutch dropped a race-prepped 911! While in my imagination I'd like to think the average 911 driver has a greater degree of technical knowhow, I'm not even sure they'd pass the pop quiz of telling me exactly what all those words meant, and certainly wouldn't be able to execute it perfectly under pressure in a repeatable way. Even with a seasoned pro behind the wheel:

Over the next few runs, as the Porsche driver honed his launch, the 911 trimmed 0.3 second off the time. The Cybertruck only posted one quicker run, which amounted to a trivial 0.02-second improvement.

His first try wasn't ideal, his technique improved over time. While for the Tesla driver, it's plug and play:

The Cybertruck launch, in contrast, is as simple as it gets. Put it in Beast mode, step on both pedals, wait a few seconds for the truck to squat on its air springs, then release the brake. With no turbo to reach full boost, no intake manifold to pressurize, no clutch to modulate, and a big, fat torque curve available from the jump, the Beast yanks hard even with 4,228 pounds hitched to its bumper.

So you put two randos off the street, or even two average purchasers, behind the wheel and there's a good chance the Cybertruck is able to get back that .2 seconds! All Tesla would have to do to "fake" the result is alter the skill level of the driver in the Porsche. No movie magic, no editing, not even having the guy pull his punches, just use a guy who isn't a seasoned drag strip driver experienced in getting the most out of a Porsche. Which is the average situation on the street!

What this tells me is that in the absurd, American Graffiti ass hypo that I was driving my Cybertruck towing my 911 to a racetrack, and a real-life 911 with a real-life guy who bought a Porsche behind the wheel pulled up next to me at a stoplight and we locked eyes and decided to race, there's a pretty good chance the Cybertruck would win unless the guy happened to be a top 1% talent. Most 911 owners are merely rich, and not talented drag racers. That's good odds! So it seems odd to me to say Tesla lied, more likely they just tested under conditions closer to reality. .2 seconds is a world on a dragstrip, but it's nothing in real life among real drivers.

But of course, why ask any of those questions when you have an opportunity to take a shot at public pinata Elon Musk?

The debunking industry so often follows this same track. Ideologically motivated, the definition of "true" and "false" are slippery, and determined more by political advantage than by reasonable interpretation. And here, as so often, if you dig into what the Deboonkers say they did, you come away from a "false" claim with more respect for the false claim than the one anointed true! I came away from this saying, the Cybertruck really is as fast or faster than a 911, even with a lot of cargo. And honestly, as I see more of them in real life, the Cybertruck is awesome. Idk that it's a practical choice, or that all the features are fully realized, or that I'd ever consider actually buying one. But fuck if they aren't distinctive, special, and as it turns out, nearly as fast as a 911 even when they're towing a 911. That's a much truthier point, in the Colbertian sense, than it is to say ELON LIED TO YOU.

I leave it as an exercise to the reader to consider whether this is politics infecting car magazines, and how this dynamic impacts much thornier debunks that are so common in the liberal press.

People get so polarized about Elon, it's surreal. I think he's done a very good job on rockets and a pretty good job in electric cars, I think he clearly has excellent business and management skills. He must be very smart. But I don't worship the ground he walks on and dislike the weird antics he goes on, some of his political preferences, how he pumped dogecoin of all things. High INT, lower WIS and CHA.

It's like China-US or Russia-Ukraine. You only see the Arnaud Bertrands of this world, people who can't make a single tweet without bootlicking glorious utopian Chinese multilateralism in this late-imperial Amerikkkan hellscape. Or on the other side there are the people who go on and on about Social Credit, implying it's something that it isn't. Making fraudsters and sleazebags pay a deposit to borrow an e-bike is not the end of the world. There's an entire genre of youtube videos full of 'China is FINISHED' 'It's collapsing' 'It's over for Xi' 'DONE' - and it's completely detached from reality where everyone is worried about Chinese overproduction. They're not exactly collapsing.

With Elon, there's this huge community that seems to think he's a complete fraudster, a cartoonishly villainous Apartheid-enriched monster who somehow tricks Muskrats into giving him billions of dollars while he lies ad infinitum. Anything he does do is purely the result of his engineers. The logical conclusion of this worldview is that the reason NASA hasn't been making great strides despite vast funding is because their engineers are garbage - sack them all. But nobody ever says that!

Great post. I think that’s what many “fact checkers” or “de bunkers” seem to miss: the world is complex. Frequently facts are hard to ascertain and untangled. Claims are conditional on certain contexts.

One can always set up something to get to the answer one wants when things are quite complex.

It not something they "miss", the complexity of the world and its associated uncertainty are what the "debunkers" are actively raging against.

Absolutely, and one can point to several fact checking site examples that oversimplify an analysis in a misleading way, but most things left-leaning media/fact checkers deboonk are indeed just complete bullshit. You can throw examples of the press repeating things like "Trump said white supremacists at Charlottesville are very good people", and I'll scoff at them with the rest of you, but I find the dismissal of fact checkers disingenuous when one considers the big picture. They're much more right than they are wrong, given how much wrongness circulates.

Sure but they're not a Trump card that ends a conversation. They're a research bureau. A lot of times if you read the whole article, even the despised deboonkers know what is going on. It's the use of headlines I object to. Mostly true, mostly false. No you need to get into the nuance of it!

For those of your following along at home, I never wound up buying myself a car. Originally I wanted to buy a new car for my wife, but she initially didn't really like the things she wanted to replace her Lexus Rx with, and wanted to keep it, so I considered buying myself a car to add something to the family fleet that wasn't approaching old age. But then her parents wanted her car, and we wanted to give it to them, so I bought her a car instead. After looking at things like the Subaru Outback, Crosstrek, Toyota Crown, etc she ultimately fell in love with the BMW 330xi. Which actually gets great marks on ConsumerReports for major reliability, and we were able to get a very low mileage lease return at the local dealer, a couple years old but in perfect condition. It's a tremendously fun car to drive for how practical it is, gets 40mpg on highway trips, and my wife loves it which is the most important thing. It's so easy to get a lot out of it, that it's turning her into more of a car enthusiast, which I love, she's taking backroads home to get to whip it around corners a bit.

And personally, I like the car a lot too, so if inshallah in a few years our kid situation is such that she wants to move to something bigger, I would have no objection whatever to driving it daily, it has a future in the family even if she moves on. Which is why i was looking at buying a car to begin with, to have one in the bank in case EVs get weird for a few years. I wish it was manual and had less complex computerization, but the paddles have come a long way so I'll live with it.

BMW 330xi

Didn't they stop making these in 2006?

No? I think my nomenclature was off though, it's 330i with xdrive, it used to be xi instead. The 30 stopped meaning a 3.0 a while ago though, it's 2.0 liter turbo mild hybrid.

Ah, okay. The 330xi seems to be a model of years past.

https://www.carfax.com/Used-BMW-3-Series-330xi_t1477

It's really just a model nomenclature, a 3er with a 3.0 liter engine and x drive used to be 330xi.

Now a car with x drive isn't in the model name, and the number is just ordinal 20<30<40 in power but not related directly to size.

It's still a 3er with appropriate power and all wheel drive, which is the core concept.

A Trip to the Mall and our Society-Wide Experiment in Extreme Trust

OR

Whatever happened to dress codes?

TLDR: We expect the vast majority of shops, restaurants, and other common commercial services to provide service to anyone regardless of appearance. This is a nearly unique experiment in human history, an effort towards not just a high-trust society but an extreme trust society, not long ago it would have been common to refuse service based on appearance. This should be considered when debating the role of trust in modern American society: we have removed the mechanisms by which one can establish trust at a glance, and as a result any degree of trust must be universally extended.

My wife's birthday was this week, and for various reasons my original birthday gift for her fell through, so instead I took her shopping at our fanciest regional mall. Which in practice meant wandering for hours through various luxury brand stores, where she mostly bought nothing but tried a lot of things on and took notes for later second-hand online shopping. What struck me most about the experience, along with going to several rather nice restaurants recently for various occasions, was that people don't dress up anymore. Not just in a general, people have no class anymore kind of way. But in a particular, we don't use dress, appearance, and presentation as a basic credit check kind of way. In the old days class was very easily visible from dress, many historical societies carried sumptuary laws forbidding certain forms of dress to the lower classes. White collar and blue collar and redneck, rather than merely being colorful phrases, were specific references to particular modes of work-clothing: a white dress shirt indicated office work, a blue denim workshirt indicated proles, a red-neck was a poor outdoor laborer with no collar at all, sunburned from labor in the fields. The presence of these class indicators showed what kind of work you did, and showed that you had the wealth to keep these things clean. And in social and commercial settings, a person in one mode of dress would be treated one way, a person in another mode of dress treated another. This has melted away.

I mean, obvious, right? But I'm at a store where the cheapest pair of shoes is $800, or a purse is $2,000, or a jewelry store with a selection of $8,000 watches. And people come in wearing flip flops, sneakers, shorts. And the sales staff were taking care of them as customers. It's summer, so of course people were dressed like that. One obvious objection is that the branding on some of those items indicates to the trained eye that a pair of flip flops can cost vastly more than any suit I've ever owned. But the staff weren't discriminating on that basis either: my canvas sneakers were Amazon chinesium, and the T shirt was Kirkland Signature, and at Ralph Lauren the salesman helped me try on a $2500 suit without blinking. The staff essentially treated, and certainly was expected to treat, everyone who came in as a potential customer regardless of presentation and appearance. I'd imagine there's some level of filth or obvious poverty that would potentially disqualify a person and lead to their being asked to leave, but I didn't see it happen. Certainly, many customers came in wearing clothing that would not reliably indicate an income over $100k/yr, and were treated with respect as potential customers. This is a remarkable fact about our society!

We've decided as a society that classism, most frequently enforced on a commercial level through dress codes and similar mechanisms, is Badtm. We all dress like slobs, and you can wander into Cartier in shorts and a T shirt and expect to be allowed in. Restaurants almost never refuse service based on appearance or dress. This is particularly a problem for Restaurants. Where the worst a bad customer can do in a retail store is steal, and this is fairly easily prevented in a luxury goods store by providing security and limiting access to product without a salesman nearby; a fancy restaurant is essentially giving you a very short term loan, giving you the goods up front and expecting payment after the meal is over. A person who refuses to pay, or leaves without paying, could in theory be arrested or sued in small claims but in practice I've never even heard of such a thing. Yet even the fanciest restaurants I've been to recently have no dress code, no attempt to screen in the most basic way that the people coming in have the ability to pay. There's no effort to screen against lower class people coming into a store or restaurant they can't afford.

Racism was, of course, the most commonly enforced form of classism until at least the 1960s. Black people, and immigrants of all kinds, were typically poor, and so if you lacked white skin or had an immigrant accent, you would be refused service. That has been eliminated, largely through long legal and social efforts by activists, but also simply isn't that useful today. I'm not sure the crowd overall was quite majority-minority, but certainly black Americans and Chinese immigrants (or tourists) formed a strong plurality among paying customers, and a definite majority of customers I saw spending vast amounts of cash on large hauls. You hear stories today about black customers having difficulty getting help, or being followed around, but I saw lots of black customers being served, and if it happens at all today it is much more subtle than one would expect if it were being used as a screening mechanism.

But I'm curious as to how and why we abandoned any effort to screen for class or presentation in these situations.

Clearly the lack of screening "works." In the sense that these stores are open and don't do it. Perhaps it is my Wawa theory of societal honesty striking again: there are few enough problem customers that you gain more from refusing to screen than you lose from screening, and that says something about our society in itself. Or maybe we're missing out on what a truly great public retail experience could be if it were done? There are a handful of boutiques that are appointment only, and restaurants at which one has to Know Somebody to get a table, and those are an obvious cuts above. But even the wealthiest wear Hermes and Rolex as status symbols, and those stores didn't really screen at all. So maybe it's a solution in search of a problem? Americans are generally honest enough that it's not worth checking.

But it's still noteworthy that this is an unparalleled experiment in human history, a society that does not discriminate based on class when providing public services, except at the extreme high end or when someone is visibly disordered. And I'm not sure what that means. I've talked before in the Wawa post linked above, about the evolution of their ordering system. At first one ordered, paid over at the register, your order slip was stamped, and then you handed it to the staff in exchange for your sandwich. Then it was that they didn't collect the slip. And now it's that most people order online, and they set the hoagies and coffees on a big rack and you walk up and take it and leave without talking to anyone or being observed or checked by anyone.

It bugs me, because I read all these screeds, from Op-Eds in respectable newspaper weekend editions to NrX substacks to published sociologists, and they all tell me that our society is becoming ever lower trust. That people don't trust their fellow citizens like they used to. And this seems intuitive to me in my day to day. But then I zoom in on some of these activities, and what I'm seeing isn't lower trust, it is higher trust. Once upon a time if you walked into a Cartier in a T shirt, they'd ask you to leave and not waste their time. If you tried to get dinner at a $100/entree restaurant without a blazer not that long ago, they would refuse to seat you. Today, we don't do that kind of screening. That's a level of trust that you see, that is manifest, and it is raised, rather than lowered. The salesman trusts you not to waste his time, the hostess trusts you to pay your bill. Perhaps they screen in more subtle ways I'm not picking up on. But they once used far more obvious ones.

And I'm not sure why they abandoned them.

Counterpoint: it is becoming more common for grocery stores & gas stations to lock their bathrooms. This is a downscaling of trust.

Maybe retail is more accepting to take money from anyone regardless of appearance. But the real trust is if they'll let you take a shit like a civilized person.

Yeah, as someone who commutes via public transit and who walks a lot, the lack of publicly-available toilets is a massive hindrance to my life, and is nearly entirely a result of the fact that homeless people cannot be trusted not to make those bathrooms filthy, or not to use them to shoot up drugs or clean themselves. When I visited Japan, I was blown away by the number of publicly-available toilets - surely a sign of the high trust level of the society. (As well as the generally small number of homeless people in that country.)

Last time I was in Japan I was using a public restroom in a park in an apparently rough part of Yokohama. The soap was a bottle on the counter with a handwritten note "please don't steal the soap" (in Japanese ofc).

In Basel, I remember when the downtown public restrooms got considerably upgraded! Automatic doors, light soundproofing, self-cleaning toilets, etc.

Not sure where they stand today, but I think this initiative would be completely wasted in most major US cities. I'm sure there were still junkies shooting up in them, but that activity was often traceless.

Public toilets in Switzerland, certainly in the big cities, are infamously drug injection/use sites, especially in Basel and Zurich. Occasionally they make an effort to clean them up, but I’m skeptical it lasts long.

Oh yeah. I remember the needle bins in the public parks, and the permissive attitudes regarding the use sites. And yet, somehow I never encountered somebody shitting in the streets, passed out on a bench, or going schizo at a random passer-by. The mentality seemed to be "Fine, you can do those things. The moment this starts getting ugly or impinge on anybody else, you will get hauled off. Keep it invisible." I saw more social dysfunction from imported Turks and Albanians than from the junkies.

Not sure how well this model has survived. My experience was about 20 years ago. But I had the sense that this only possible as a result of 'Swiss Culture', if that's not too vague. Being a smaller country also helps. It seemed to work well enough for them, but if anybody pitched the idea needle bins in the nearby parks around me here in the US, I'd tell them they're crazy.

Yes, this has been a pretty big annoyance when taking children into city downtown areas, especially. It isn't trivially easy to locate, then walk to, then order at a coffee shop or something, then get and remember the code in time.

Counter-counter point: free tap water everywhere, and at least our toilets support flushing toilet paper (looking at you, Mexico)

Comparing yourself with a third world country is not the W you think it is.

I don't think the dress codes were ever to separate thieves from decent people. When we had them, if you walked into a fancy restaurant without a jacket, the response might be "Did sir forget his jacket? Step over here, sir, and we'll get you a loaner". The dress codes were to maintain a certain atmosphere. Now that few customers care, they've largely fallen away. And in the case of some segments, fallen away from the top before the middle. When my wife and I were looking for her engagement ring in various stores, dressed as one might would for a trip to the mall (i.e. casual) we got the snooty treatment from several stores, but NOT Cartier (which is indeed)

Things like placing the orders out and unguarded work because most people just aren't casual thieves. People who steal from stores generally aren't doing it out of opportunity; they are going in there to steal particular stuff, and customer orders (especially perishable ones) aren't useful.

Trust isn't entirely unlimited; if it's violated it can be withdawn. The Jersey Mike's (a sub chain) in my area used to put the takeout orders on a table, now they're back behind the counter, I would assume due to complaints of orders disappearing.

The Jersey Mike's (a sub chain) in my area used to put the takeout orders on a table, now they're back behind the counter, I would assume due to complaints of orders disappearing.

>Mike

>from New Jersey

>calls his sub chain Jersey Mike’s

is he stupid?

Over the recent years I’ve seen quite a few of my go-to restaurants go from self-serve unsupervised online order pickup shelf, to semi-supervised pickup shelf near the cashier or other employee, to pickup from behind the cashier or employee handoff only, or to even no pickup anymore at all.

It was great while it lasted, being able to swing into a place to pick-up my meal and swing out without breaking stride.

Maybe like a #JustGirlyThings quote, I shouldn’t be sad that such a phase happened, but happy that it happened in the first place.

And I'm not sure why they abandoned them.

There is an extremely obvious answer that jumps out at me from reading the text - discrimination laws. Even if you just want to keep out the riff-raff and the poor, class-based policies like the one you're suggesting are going to be an absolute goldmine for any lawyer who knows what the phrase "disparate impact" means. A policy which keeps out members of the societal underclass is going to disproportionately impact black people, which means it is then going to have the business which upholds that policy wiped out in court if seriously challenged.

Today, we don't do that kind of screening. That's a level of trust that you see, that is manifest, and it is raised, rather than lowered.

I actually disagree - there is in fact less trust. What happened is that the spread of insurance and large corporations mean that the costs of accounting for those problems that you're talking about are simply spread out and distributed across the rest of society and the rest of that corporation. They aren't trusting you or their customers - structural changes mean that there's just not really anything you could do to seriously inconvenience them. If you go into an Apple store and just wreck the entire place, destroying/stealing every single piece of tech in there, the costs of your actions aren't going to be added to the bills of people who shop there - those customers are already paying for that risk and have been for years.

There’s periodic discussions of one, particular, item commonly on dress codes.

Sagging. For those not in the know, it’s a (very black coded) fashion in which the pants sag down enough to reveal the undergarments, or these days often basketball shorts worn underneath regular pants. I see plenty of ‘no sagging’ signs at businesses, although not usually high end ones(those people wouldn’t go to such anyways). This is sometimes controversial because it’s black coded, but a) no one wants to see it and b) blacks who object to sagging are reliably of the better sort while whites who engage in it are reliably trailer trash. A few municipalities have carried out campaigns against sagging- Dallas ran a series of billboards with the slogan ‘big mama says- pull ‘em up’ a while back. These campaigns are routinely mocked by people who, themselves, view sagging as uncouth ridiculosity.

What I think gets left out is that clothes send a message and the act of sending a message is one which reinforces the truthfulness thereof. In a certain sense the spread of the sagging fashion convinces people who do it to act more like they have neck tattoos, and jeans and a ball cap convince people to act more salt of the earth, and dressing like a harlot convinces a woman to act more like one. All of these statements are controversial because there is a truthfulness to them; our fashion choices are conformity with other people who follow the same fashion choices.

I feel like the obvious explanation is that clothes (at least such obvious ones) ceased being reliable indicators of the things they would want to screen for. I know this is partly my cultural milieu (west-coast-tech-types) but I basically never see a suit in the office. Or on the street. Or almost anywhere that isn't interacting with some financial services vendor or high end retail. I wear a suit very rarely (generally when a restaurant dress code calls for it) and pull down a pretty comfortable income. Before wearing certain kinds of clothes can be used as an effective screen it has to be an effective signal and I think this is mostly not true. Largely as a result of wealthier people dressing down.

I manage a Men’s Department @Dillards and wear a tie, vest, dress pants, etc

I look like a fucking salesman .., which I more or less am.

It’s just not a thing at all in the lower or middle classes.

Interesting observation.

And I'm not sure why they abandoned them.

Probably for barber pole of class signaling reasons, combined with physical objects like suits becoming reasonably cheap and accessible to the working classes.

I don't really know people with nice handbags or jewelry, but for the kind of store that has representatives in malls, it seems to be at least as much a matter of motivation as class. Tradesmen can and do buy $100,000 trucks and $500 boots, and would probably buy their wives some nice jewelry or a nice bag if they really wanted that. They might be more likely to just walk into a store and buy the thing than someone in a higher social class, but who isn't embarrassed to take notes and go look for a better deal online.

Customer service people probably can tell underclass and teenagers likely to shoplift from body language and speech patterns more than by clothing. That doesn't necessarily suggest higher trust, simply that the class markers have changed.

Great post, interesting observation, keen to hear what others have to say. But two quick thoughts.

(1) how much of this is the rise of wealthy Arabs/Russians/Chinese etc. as potential potential customers? Norms can be difficult to communicate cross-culturally and even harder to motivate (“so what if French people think shorts are just for sport and the beach, if it’s hot I’ll wear them to lunch”). But as the purchasing power of outsiders increases, the cost of excluding them becomes greater, so these codes get retired.

(2) Enforcing these codes requires a certain amount of skill and perspicacity, especially once we get beyond Rolexes into Patek Philippes and Vacheron Constantins. As the role of sales clerk has been shunted down the social ladder, most employees don’t have the knowledge or empowerment to enforce them.

On (1), it's probably people from other regions of the US as well. Especially California, but the entire West has been informal for multiple generations now. When I was a child, men could dress up for going out either Hawaiian or Texan, and the women would wear their normal dresses, but add some artisinal turquoise and silver jewelry. My family usually dressed Hawaiian -- you can just wear shorts and sandals instead of needing nice boots and maybe a nice belt buckle as well.

Most sales assistants at the very top of fashion and jewelry are either rich kids or wealthy older women looking for light work after the kids have left home. Quite a few are gay men, usually of upper middle class origin, who can make $200k a year doing very little real work other than hanging around the store and bitching all day in between flattering customers. I wouldn’t really describe any of them as particularly low on the social ladder.

Clothing and fashion lost its value as signifiers.

Dressing a certain way sends certain messages about what you do, who you are, what tax bracket you're in. However, as new money piles in and out, demographics shift, and things shift on the timescale of days instead of years, the symbols lose their meaning. Society becoming lower trust is a side effect of losing the meaning behind the symbols; you used to be able to trust that the well-dressed man in a suit is on his way to a white collar business job. Now you're not sure if he's a crackhead or a psychopath, or some combination of all of the above.

When all the tech CEOs took fashion lessons from Steve Jobs and influencers pulling millions a year drive Lamborghinis in cargo shorts, when scantily-clad women are not 'asking for it' and how dare you imply such a thing, when nouveau-riche Chinese cover themselves in brand names and gold, then why would you bother dressing a certain way if it's fundamentally interchangeable, meaningless? Fuck that, people will wear what's cheap or comfortable in the end.

Might also be worth looking at the "stealth wealth" trend, and the churn in fast fashion.

While it might not be a good signifier of wealth it does really impact your appearance. Well fitting clothes, better materials and wearing something nicer than a t-shirt can easily add two points on a scale from 1-10. Attractive people are more popular, are perceived as smarter and more moral, have better chances at attracting and retaining a mate and are more financially successful.

If you are really wealthy, you do not have to care how you appear to the plebs at all. The middle classes must care about their appearance, bums and billionaires are free.

I just want to thank you for giving a headline, a sub-headline and a TL;DR. Good posting form.

I try my best. No one wants to read all that.

Personally, I absolutely love this change and hope it persists. There's something so charming about for example a famous person walking into your store and still introducing themself with something like "hey, I'm Rob, nice to meet you." First name basis with people, more equal treatment, it's not even purely about trust per se, though you do bring up a good point about it. It's the logical continuation of the American disdain for titles and kings. Frankly even if I met someone who was knighted, I'd refuse to use Sir on principle, because I love that about us. To adapt MLK, "I have a dream that one day our children will be judged not by the brand or quality of their clothes but the content of their character." It's freeing. Just like when you realize that the rule of "it's not awkward unless you make it awkward" is incredibly powerful, and you can have difficult or sensitive discussions with people without hiding behind taboo, it also is liberating. As I like to say, people are just people, so the less we do to hide and obscure that fact, the better and kinder I think we are inclined to be.

I dunno. This doesn't check out IME. I think a bit of masking is actually necessary. There was an appeal to getting more intimate with the minds of others when I thought "people are just fundamentally good". While it's not like I believe the opposite now, I can't sign on to that statement as-is.

I have been low-key horrified at many of the utterances good friends and peers have made over the last decade. These are decent, nonviolent, funny human beings who turned on a dime and started expressing every cruel, nasty thought they had in the name of authenticity and 'speaking their mind'. I have not cherished this. I wish they had actually shut the fuck up and kept it to themselves. Our bonds were not strengthened, but frayed. They still are to this day.

That's with actual people I'm familiar with in my life. You can probably imagine how much more insufferable this is with celebrities. More irritating is how that class is allowed to express their 'authentic selves' as much as they want while being completely shielded from the consequences of their expressions, while others get no such protections. Pedro Pascal should be C-tier after all the crap he's said, but instead he soaks up more love, more accolades, and more roles. I guess him and his legions of fans are okay with this, and who am I to complain. But I do not feel inclined towards kindness at this state of affairs.

Maybe society needs masks, and kayfabe, and to be just a little fake and gay. Maybe we are better off with some illusions regarding others. Because just like with the global adoption of the internet that was supposedly meant to help us better understand each other - well, it worked. And I am thoroughly displeased with the results.

See Destiny and his ongoing meltdown. I actually thought he was closer to being 'one of the good ones' worth listening to occasionally, and he seemed to make a concerted effort to drop some of the low-effort gotchas that marked thie beginning of his career as a political streamer. Now I think he's telling us what he really thinks. While I believe this is somewhat of a public good, because now I know I don't ever have to pay attention to him again, it is still depressing and unfortunate. And I also now have in mind several friends who are exhibiting this same behavior to a lesser degree, which makes it doubly so.

I mean, humans being what they are, it's more like we need to give people a chance to be good. It may not happen automatically. I think part of that is searching for common ground and, maybe not values exactly, but starting conversations from a similar point. However, it's definitely tough out there. I had a conversation last month with my brother who I was absolutely shocked to see almost explicitly advocate for rage and violence as necessary to wake people up and get people moving (he is very pro-Palestinian). I still think and worry about that, frankly, radicalization and extremism. I was like look, MLK got civil rights done at the end of the day, not the Black Panthers. He still sort of thinks that whites needed to be 'scared' into it, but I strongly disagree. It was getting moderate whites on board by emphasizing our shared humanity and showing a human face to the suffering. Things like Selma, you know? Hard to ignore.

So we ended on what I felt like was at least an okay note, because I ended up saying hey look, I lean Israel here but it's fucked up all around and just a bad situation. But one thing I do feel strongly about is Palestinians are straight up not getting enough food to live. That, IMO, is and always will be on Israel, who controls the borders - it's not like Palestine can feed itself, and huge chunks of farmland were bombed or bulldozed or what have you anyways. So I'm like hey, we feel powerless and that really sucks, let's do something together and call and email our congresspeople, who actually do have someone read/listen to those. It's a small thing, but felt nice, and was something we were able to come together on. But still, it does still really suck and I get that. I really don't like seeing that kind of attitude so close to home.

Well I got a bit off topic but I don't see casual, equal, class-blind service and conversation as really posing too extreme a risk of people indulging their worst selves instead of putting their best foot forward. Aren't most of these mores really about respect and treatment of people short of friends, not friends per se? I think there's still some rules of politeness involved, it's just a casual politeness and not a formal one.

I was like look, MLK got civil rights done at the end of the day, not the Black Panthers. He still sort of thinks that whites needed to be 'scared' into it, but I strongly disagree. It was getting moderate whites on board by emphasizing our shared humanity and showing a human face to the suffering. Things like Selma, you know? Hard to ignore.

Yeah sorry, whatever your brother said about how the “Civil Rights” movement won its political gains is almost guaranteed to be more historically accurate than the extremely sanitized, simplified, mythologized version you’ve presented here. If this is the narrative you need to believe in to allow yourself to decry political violence and seek conciliation, then by all means please continue to believe in it. But it doesn’t actually bear much resemblance to the nitty-gritty details of how that particular sausage got made at the time.

They all played some part of course, that’s just the sausage of history indeed. But man, the 60s were ugly. BLM and a single assassination attempt is tame by comparison. Apparently the message LBJ used to carry the portion of Southern senators needed to break the filibuster was the basic idea “better you now than someone more radical later” - so the framing of people seeing some sort of racial equality effort being law was seen as inevitable, make of that what you will. But there’s at least some mainstream thought such as some research here including citations in the intro that suggests nonviolent protests were associated with both successful campaigns and shift in vote share more often and more strongly than violent ones. Of course, a funny fact is that at least per the polls, a good chunk of people thought the March on Washington even was counter-productive. MLK wasn’t actually super cuddly and moderate, he was dedicated to making whites feel uncomfortable, but there’s a difference between that kind of “troublemaking” and the more violent kind, even if you might plausibly call both radical or even maybe extremist.

But at the end of the day it was white politicians giving more advanced civil rights to Black people.

But there’s at least some mainstream thought such as some research here including citations in the intro that suggests nonviolent protests were associated with both successful campaigns and shift in vote share more often and more strongly than violent ones.

Nonviolent protests, especially back then, ran under good cop/bad cop where the violent protests made the nonviolent ones effective.

There's also the fact that "nonviolent" and "doesn't cause harm" aren't the same thing. Protests in the 60s were absolutely meant to cause harm to members of the outgroup. Telling your employer that you tweeted in support of assassination is a nonviolent protest (and so is firing someone for that tweet).

I think it’s a negative thing. I think that a loss of respect for yourself and others is often shown by how we present ourselves in public. When you’re dressed well you treat yourself as a person worthy of respect and treat the rest of society as aplace worthy of being respectable for. When men wore suits it wasn’t just an empty signal but came with a statement of respect for others. A guy in a suit insisting on being called Mister and calling his boss Sir or Mister or whatever and who is teaching his sons to treat themselves as people worthy of respect and to respect others is contributing to a lot of very important and beneficial things for society at large. The practice of demanding excellence from ourselves and respect from other works to create a society in which excellence and respect are norms and that even those at the bottom of the social ladder.

When the rich choose to forgo those things it encourages others to do so when they can least afford the problems that come with it. A rich person can afford to talk back to his boss because he has enough cushion to weather a job loss. A rich person can be loud and proud about vices like drug use or drinking or casual sex because he can get access to things to fix any problems that come up. This often leaves a wake of people behind who emulated bad behavior without the means to avoid the consequences.

The other thing that happens is that it erodes the culture’s ability to demand good behavior. We lose the standard and the ability to enforce the standards. When you don’t feel the need to dress appropriately for going out, you also can’t say much about others taking it farther. You can’t get that mad about the people wearing pajama pants to the grocery store when you’re wearing sweatpants. You can’t say anything about being lazy when you’re lazy.

Tech workers can be very wealthy and dress like shit. Poor people can be very poor but look wealthy, either through knockoffs or through ill-advised purchases. While physical attire isn’t being judged, the workers are probably checking your social cues to gauge level of wealth. You can tell the difference between a tech worker and a hillbilly even if they both wear cargo shorts.

don't use dress, appearance, and presentation as a basic credit check kind of way

at the commercial mall. Various subcultures still judge your appearance. From youth subcultures to finance. People dress better at industry conferences and on instagram. The mall is just no longer a place where any social encounter of value transpires. It’s a dead third space.

in theory be arrested or sued in small claims but in practice I've never even heard of such a thing

This does happen. Maybe not in San Francisco or NYC I guess, but elsewhere the restaurant will send the video of your car to the police who will charge you.

indicates high trust

I think it indicates a breakdown in predictable attire signaling. There’s just tons of wealthy people who don’t dress up. They can be billionaires and they won’t dress up. There can be people who dress up but waste the attendant’s time. And then of course there’s the prospect that the wealthy person you’ve turned away for looking poor goes to the news or your manager or Twitter. I wouldn’t say it indicates high trust, but alienation from a useful common language of socioeconomic signaling.

Your link appears to be broken for me, goes to an unlisted video of "youtube is not supported on this device".

It's not a real video url, just site plus timestamp. Most likely he tried to copy-paste in the timestamp and overwrote the video id. I do that all the time trying to hand share YouTube vids on phone.

Fixed. Just a silly link from a recent comedy episode (the Shane Gillis Trump impersonation is insane, though)

Same, an unusual error I don't think I've seen before.

It doesn't look like there's any video link, it's just youtube dot com slash watch and then the timecode.

I feel that. Im travelling in asia right now, and one thing that jumps out is how well dressed people are here compared to back home. Im kind of embarrassed by how shlubby most of the white tourists look here. Ratty stained t shirts vs suits or stylish street wear.

I think it does help with social cohesion. People here are a super high trust, low crime society. Shops leave their wares unguarded on the street, and nobody steals it. Of course there's many factors for that, but i have to think that a society where everyone dresses terribly helps to erode the social fabric.

I would guess its a combination of car and digital culture. We're so alienated from each other, we just dont see each other much in person. In a more traditional society where people still socialize and conduct business face to face, clothes matter a lot more.

I also tend to think most of the modern clothes sold to men just suck. Its either hip hop, video game graphic tees, or gay country club shit. Very little to make an adult straight man feel cool.

To your point of people in Asia dressing much better than their Western counterparts - absolutely. I was shocked by how stylish the Japanese and Korean people were last time I visited. This ties into your last paragraph too, the big brand stores like Uniqlo, Muji etc, make excellent street wear that is really affordable and lacks the garish logos and brand names all over it that you see in hypewear (BAPE, Balenciaga etc).

Doesn't remotely apply to China however, though a lot of this is probably down to the lower level of wealth and presence of many more poor people, even in the tier 1 cities.

One thing I like about living where I do is that at least my rich largely Arab, Russian, Continental European and Chinese neighbors know how to dress better than most modern Angloids. Super rich Arabs of the non-niqabi kind often have a very pleasing kind of modest-Jackie-Kennedy-meets-Loro-Piana vibe (of course Loro is now trashy, I assume it’s actually Franck Namani), with very good understanding of color and a palette that usually stacks cream and white with the occasional burnt orange or blue accent.

Wealthy young people certainly still judge each other on their clothes and spend (tens of) thousands of dollars trying to look cool. That their choices are often extremely ugly is a sad indictment of zoomer fashion, and perhaps fashion itself, but I certainly don’t think it means people no longer judge each other based on their clothes. It’s more likely that sales assistants at expensive US regional malls know that there are plenty of schlubbily dressed people who come in and drop $50k and adjust accordingly. But those people aren’t cool, or even high status, necessarily.

Yes, most outward marks of class vanished, but class remains. In classless society, there would be no such thing as luxury shops and luxury products.

What happened is that class is disguised. Your boss is not dressed in silks and adorned in gold any more, he does not demand you bowing and scraping before him any more. He looks equally shlubby as you, he shakes your hand and calls you by your first name, but is still your boss.

But I'm curious as to how and why we abandoned any effort to screen for class or presentation in these situations.

Clearly the lack of screening "works."

Clearly, modern capitalism works much better for the upper classes.

Remember, in trad sharply dressed society 100 years ago, lower class dissent and revolution was omnipresent threat. Massive strikes, riots and urprisings were commonplace, revolutionary parties demanding expropriation of the capitalists had mass support.

These things are unthinkable now, the rich are simultaneously richer than ever before, and more secure than ever before. No one in mainstream politics is any threat to them, and they know it.

I am not saying it is because lack of high fashion, but this can be part of the explanation of this mystery.

But even the wealthiest wear Hermes and Rolexe as status symbols, and those stores didn't really screen at all.

You can try on a Hermes or Rolex, but you can't buy one. You have to do everything right to get that call.

You can just straight up buy almost all Rolex watches and almost all items at Hermes too (excepting stuff like Kelly bags but even then you can buy them post retail for a markup).

Plus Rolex is the poor man's idea of a high end watch brand. Their stuff is overpriced and honestly, somewhat declassé among those who are actually interested in horology.

Plus Rolex is the poor man's idea of a high end watch brand. Their stuff is overpriced

Vicious propaganda spread by AP, Patek, and RM fans. It is perhaps more correct to say Rolex is a rich man's idea of a "blue collar" brand. In horology circles, Rolex is the Toyota Camry: economical, ubiquitous, not particularly flashy (compared to other options) or complicated (in the horological sense), and the high end models are mostly a silly joke and a waste of money. But your basic-bitch Submariners, Explorers, and GMT masters remain about the single most accurate, reliable, and abuse-tolerant mechanical watches sold today.

At one point I had finished some consulting work and was considering getting a moderately fancy (few hundred dollar range) watch. My wife warned me that I should on no account get a Rolex. She ended up taking charge of the project for me and got me a nice vintage Seamaster.

My daily driver is a vintage (well, 1990s) midsize Seamaster! It's actually the same watch Joe Biden wears. It's a quartz because I can't be bothered to reset it every few days if I'm not gonna wear it which takes away some of the magic but I have other watches for that kind of show that stay in their box unless I have guests over who would be interested in seeing an IRL tourbillon etc.

Btw, this is amazing for learning how a standard mechanical watch works. The animations are top notch.

I knew before I previewed the link that this would be the ciechanowski animation, and I'm here to express my extreme delight with everything on that site. The GPS animations are also stand-out spectacular. One of the few people on the internet doing interesting things with the new WASM toys.

That man seriously deserves a Public Engagement in Science/Engineering award. Too bad that the people who usually get those kinds of awards aim their content at the level of an average 8th grader and not an intelligent adult who knows a thing or two about STEM...

I thought I understood how bicycles work. Then I read his bicycle article...

This is an odd comment. Rolex AD's are notorious for stonewalling customers who wish to buy (particularly newer models) and making them jump through all sorts of hoops. By this I mean retail authorized dealers, not gray market. And there's nothing at all "poor man" about Rolex movement or quality, even for the Vacheron Constantin or Richard Mille crowd.

You may be right that it's a well-known brand and is by no means at the apex of watch pricing, but Rolex isn't in the category of mall fashion watch quite yet.

Why you shouldn't buy a Rolex

An interesting video. He makes an off-hand mention to the high-pressure sales tactic of 'receiving the call' and talks about the various negatives of having one.

One thing I'm surprised about is that they're apparently... finiky? As in, need regular maintenance. Rather disappointing. Give some of the videos of watch restoration of Rolexs that were worn as a daily beater for decades, I wonder if that's a more recent development.

Still, having one isn't on my to-do list, ever. If I wanted a high-end watch, I'd just buy an Omega Speedmaster and be done with it.

Why you shouldn't buy a Rolex

"I own three Rolexes" --> makes a video on why you shouldn't by Rolexes.

I agree with most of what he says about the price-gouging, luxury branding, and artificial or prestige pricing. When Sean Connery would have bought his submariner 6538 it would have cost around 2-300 bucks, which in today's dollars would be maybe 2K ish. A submariner today MSRP is anywhere from 10K-40K+. Plus the hoop-jumping for the AD. It's absurd. You can get a Rolex on the grey market with box and papers for 2K USD depending on the version (not a new one, a new one will be considerably more).

I still think they're good watches, and I've never heard about any problems with quality control as long as the watch isn't routinely abused. Certainly not in terms of mechanics. I don't love all their styles, but I would probably prefer any Rolex to, say, a Hublot, where the average price is like 20K and more often than not they look like garbage (this is just my opinion).

I happen to have a Speedmaster and endorse your choice.

Not true for Rolexes, but yes if you want a shot at a Birkin bag you need to build up a relationship as an Hermes customer.

Similar for certain Rolexes. Desirable colorways are often available on a limited application basis.

Nah, you just need to find the right store (some airports, Tokyo) and have a man walk in and buy it for you (since the assumption is that they’re less willing to play the game). Even if you do neither of those things, you can negotiate a purchase as a new customer in a single day, it’s just about how you talk to the SA. The mythos around this suits the brand, but the idea that you have to spend $100k on scarves and horse saddles before they let you buy a black Birkin is just that.

Seems to me the obvious explanation is that they are worried this will look racist. Rejecting a redneck might have been fine, but you can't reject a black person for wearing the wrong clothes without getting sued or boycotted. The only resolution is to not have dress codes.

La Perla at South Coast Plaza, one of the premiere malls in the US, had people constantly complain about haughty staff, informing them their goods were more expensive than it looked like they could afford etc. The store then left the mall. My uncle mallwalks there in ratty t shirts with holes, which saddens me.