site banner

Culture War Roundup for the week of November 28, 2022

This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.

Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.

We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:

  • Shaming.

  • Attempting to 'build consensus' or enforce ideological conformity.

  • Making sweeping generalizations to vilify a group you dislike.

  • Recruiting for a cause.

  • Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.

In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:

  • Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.

  • Be as precise and charitable as you can. Don't paraphrase unflatteringly.

  • Don't imply that someone said something they did not say, even if you think it follows from what they said.

  • Write like everyone is reading and you want them to be included in the discussion.

On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.

16
Jump in the discussion.

No email address required.

It's worth pointing out that Twitter is still fully operational two weeks after losing some huge percentage of its staff.

People on this very board were calling for near-certain failure due to key staff leaving. And while it's too early to say definitely, I think it's not too early to start updating in the direction that no, these staff were not in fact necessary to the continued operation of Twitter.

I'd imagine that the administrators of a university are even more unnecessary.

I agree that every one of these administrators aren't vital to the school's success. However, I think Twitter is the exception rather than the rule. You also have to consider that Twitter's changes are very recent and also came with a change in direction. Among other things, old Twitter valued content moderation while new Twitter does not. This wouldn't have any effect of Twitter being operational since that division didn't affect the online availability of Twitter.

old Twitter valued content moderation while new Twitter does not

This isn't true. People still get banned. The people who got fired were those who formed committees that decided on strategic bans. Here's a web of twitter we don't like, let's ban the biggest nodes to shut it down.

Harvard could easily cut 90% of its staff, as could any of these places where there aren't any physical goods and a surplus of cash.

these staff were not in fact necessary to the continued operation of Twitter.

Software is fundamentally no different than other kinds of industrial endeavor when it comes to the infrastructure. It's fundamentally a blue-collar profession for people who desperately want to pretend it isn't. Pays the same, too (boom towns skew this average though).

In both industries, it takes an order of magnitude more people to construct something than it does to maintain it, but the people who maintain it are a trade unto themselves. The software industry calls them "developer operators" (to be fair, in most companies there's nothing "developer" about them... but the corruption of the term doesn't dismiss what it was fundamentally made to describe); the real world calls them "millwrights".

The bottleneck in development is ultimately how parallelizable the work is, which is why software companies that employ a lot of developers have to grow breadth-first (it's very difficult to justify throwing 10 people at a simple API; you need 10 "areas of concern" and they all need to be slowly growing). The developer's favorite line is "like asking 9 women to make a baby in one month" for this exact reason.

However, software is unique in that the operators of the system work remotely for free. So you actually can fire every single one of your local operations staff that aren't relevant to the fundamental operation of the system and still make just as much money in a pinch; yeah, you run the risk of losing institutional knowledge when it comes to improving or modifying the system... but Twitter (nor any company in dire financial straits) doesn't need to do that, so it's the first thing to go (after the sinecures).

I'd imagine that the administrators of a university are even more unnecessary.

I take a different opinion of this: the sinecure positions are there because they're important to fundraising. If some Blue-leaning rich dude wants to change the world, well, "change the world" is what the university is selling. But they're not very good at changing the world because R&D is hard, and all the things that could make a real difference are banned, so the only thing they even have to sell is social change. And "if you can help pay our admin's salary, we'll help shape our graduates in the political direction of your choosing" is something that will get people to open their wallets.

Besides, what else are the rich going to do? Venture capital? That's risky and has the same problems. By contrast, paying people with the power to fail the next generation if they refuse to DIE has a demonstrably positive return on investment.

I believe that when you, @curious_straight_ca, and now me talk about developer operators/devops/sre, we're all talking about different things. From talking with people, this part of the industry seems to be undergoing a complete headfuck of an identity crisis.

Many companies simply changed labels from "sysadmin" or "operations" to "devops" or "sre" and called it a day, with the people doing what they always did: maintaining hardware, producing an endless stream of small automation (bash scripts, yay), and managing LDAP/other access (your next post seems to describe this group). Some companies created a true group of automation developers--tool and systems makers. Yet others have formed groups inspired by manufacturing that are using statistical modeling methods to design and maintain systems.

What I think this points to is that the industry is struggling with the problem of reliability, specifically, reliable product delivery (availability, durability, latency, etc). Up until fairly recently, no one from the industry spent time to devise a production process(1) to produce reliable software, so it's pretty much a wild west composed of at least two dozen or more different groups trying to figure it out (or make money and disappear), all of them using the same two labels.

I'm urging caution because it seems there's a strong disconnect between map (job titles) and territory (job responsibilities) to the point where two people with the same title probably can't even communicate because they do vastly different things.

(1): I'm not counting tools like formal verification or Erlang ("nine nines")--I'm thinking of the whole process including people, skillsets, organization, etc.

sre and blue-collar maintenance are different. one requires coding, understanding the design of complex systems. the other requires relatively simple manual labor.

yeah, you run the risk of losing institutional knowledge when it comes to improving or modifying the system... but Twitter (nor any company in dire financial straits) doesn't need to do that, so it's the first thing to go (after the sinecures

elon wants to make large-scale changes to twitter to make it profitable. "X: the everything app", "payments", long-form video, long-form tweets, new verified system, etc. he does need to 'improve and modify' the system.

one requires coding, understanding the design of complex systems.

Yes, that's the millwright job (if you can't handle the math to reason about them you don't get the ticket to practice it). The style of programming "language" they tend to use is also one not many conventional developers even know exists.

By contrast, Devops is generally a simple position- generally alternating between "fix the hardware and the racks", complaining that the software running on said racks isn't their department, and claiming that they don't have time to grant access to the resources development needs to do its job any time this year because reasons. (Apologies to you if that's your trade; I'm sure you're one of the good ones.)

Most software engineering for that matter is also relatively simple labor- the thing that makes it engineering is that you have to do the design work too, and make sure every component interoperates (and is even fit for the task in the first place), and if it doesn't work (or stops working) they're ultimately in charge of modifying it so it does, and the list goes on. Sure, it can be a bit trickier because everything's virtual, but the upside is that all the error messages are generally in plain-ish English rather than "it makes a weird grinding sound but only below 2/3rds power".

It's depth that's not necessarily obvious unless you've actually seen it, but you do actually need a decent amount of brains and general systems knowledge to do either job well. It's like saying "yeah, anyone could do software development because 99% of it is just implementing CRUD over and over again"; it's technically true but an over-simplification.

I'll accept much of devops like that, but twitter really does have 100Ms of users, global scale, and a realtime timeline, search, ads, replies, etc system, that have to be understood, monitored, designed to operate fail in recoverable ways despite very complex interactions of all the systems even when being actively changed. Many of the SREs elon fired weren't just rack swappers, they were coders who designed systems to preemptively avoid complex failure cases. According to him, his job started as you say, but that changed -

When I joined the team the first project I had was to swap old machines that were being retired for new machines. There were no tools or automation to do this, I was given a spreadsheet with server names. I am happy to say operations on that team is not like that anymore!

And then

We had bugs where new cache servers wouldn’t be added back(race condition on start up) or sometimes it took up to 10 minutes to add a server back(O(n^n) logic). Since we weren’t bogged down by manual tasks thanks to all of this automation work we could develop a culture in the team where we could go and fix these while keeping projects on track.

Sure, it can be a bit trickier because everything's virtual, but the upside is that all the error messages are generally in plain-ish English rather than "it makes a weird grinding sound but only below 2/3rds power".

The actual error is sadly only rarely actually pointed out by the error message. Compiler errors are straightforward, runtime errors are not, and runtime errors in complex interconnected systems may as well be Kafka's Prozess.

And the largest category of errors doesn't even have messages.

Fully agreed on DevOps though.

True, that's why it's plainish English.

Hell, even the compiler errors aren't straightforward the less structured the language gets; it's even better when said compiler does shit like "you meant for that object to not have its keys named the same thing, right? I'll just do it automatically because... fuck you, I guess".

It's a mess I'm not convinced even AI tools will be able to solve, so maybe I should just be thankful for the job security.

In what language do compilers do that? The horror!

So you actually can fire every single one of your local operations staff that aren't relevant to the fundamental operation of the system and still make just as much money in a pinch; yeah, you run the risk of losing institutional knowledge when it comes to improving or modifying the system...

And that's why the claims about Twitter going down just made no sense to me. If the engineers were good and they made a stable system, they would not be necessary for the continued stability of that system. If the engineers were bad and made an unstable system, then good riddance surely.

However, on the issue of improvements, I'd also like to note that, as a Twitter user, the rate of improvements to the system has actually increased since Musk took over. What I'm not seeing is a lot of people updating their priors based on Musk's success. Perhaps it's natural to stick one's heads in the sand and think "not me - I am necessary".

If the engineers were good and they made a stable system, they would not be necessary for the continued stability of that system

this is just wrong. one of the big reasons why is scaling and new features are very significant changes to the system. twitter would need much less reliability work if there wasn't any new code being deployed, or anything being scaled up. but neither of those are true.

What I'm not seeing is a lot of people updating their priors based on Musk's success.

They all have financial (a business' bad heuristics as to your necessity is to your advantage) or political (wokes mad their jannies aren't in charge no more) to pretend they don't know (or exaggerate/lie if they do).

Why would they speak out to their detriment?

(Meanwhile, the people taking advantage of this information are not posting about it, either because they're preparing layoffs of their own, because they're trying to acquire labor at a relative discount, or because they're too busy otherwise.)