This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Having worked as a refrigeration tech before, I may have some insight- this was almost certainly done for management and monitoring. Defrost cycles are very important to keep freezers from bricking themselves, and danfoss controllers are very common as an alternative to a mechanical time clock- because they’re fully programmable to the conditions of the specific box.
Why they were connected to the internet is probably because corporate wanted to be able to mess with it without sending someone in person. The stated reason was probably so they could set passwords but in reality it was to make in house maintenance personnel’s lives easier.
There's a lot of value in things like alerts, too. Being able to send someone an text message or even call someone when a glycol loop herps the derp can be a 10k-100k dollar value proposition, and pulling good logging data about current draw can give you warning months before actual death about a failing motor or compressor.
That said, the idea of inbound open ports for management is mind-boggling, even for hardware in the 2000s era. Even before you get to the security ramifications, it's just an absolute nightmare trying to get fifty sites to stick with static IP addresses, inform you when changing ISPs or modifying firewall configurations, or your DDNS provider changes literally anything. I know OT has worse practices than IT, but this one is pretty mind-boggling.
Forget setting up VPNs. Buy and configure a single main operational server, have the hardware send regular updates there, move all the heavy lifting and the UI somewhere you can control if and limit firmware updates to message protocols, certificates, and fundamentals. Allow big customers to buy, run, and redirect to their own server if they really need to own everything. Done. Two or three months work for an intern. I hate Everything Is The Cloud, but this is a tailormade situation for just putting things on the cloud.
Depending on the way the corporate network is setup, you might need one of these at each location: does a grocery chain have a corporate-wide VPN already? I assume not because then you could avoid this whole situation by never exposing the hardware to the Internet. Even then, locking down odd hardware in a subnet would be a good choice.
It's often said that the "S" in IoT stands for security, and it's not really wrong, but the way we define "security" does manage to be difficult to implement on embedded devices. "Oh just use TLS" isn't trivial: most configurations there will need certs that get updated regularly. Do you even trust your CA to remain on the approved lists indefinitely? Will your choice of RSA-4096 still be acceptable for the life of the unit (say, 50 years)? What's your plan for post-quantum cryptography on a device built in 2003?
Your fridge also suddenly needs a good clock source to check the certificate time limits: now you need to set that on the device, potentially every time it loses power. NTP works, but isn't foolproof or even secure in most deployments. These things aren't trivial, unfortunately.
In practice, a large, long-lived SSH key signed by corporate would work fine, but none of the normal tech stack is really setup for this today unless your users like command lines. I'm not saying it can't be done, but it's not in the normal wheelhouse for technicians who specialize in wrenches.
I've never worked for one, but I assume they do for the POS system, along with other ERP system access for inventory, receiving, accounting, and so on. They should probably have multiple subnets per site for the sake of isolation, but that's a bit of a hassle to combine with a VPN and may not have been done.
More options
Context Copy link
More options
Context Copy link
They had inbound open ports to allow their maintenance managers to change settings store managers were messing with, probably. But why they didn’t take security precautions to prevent that I don’t know. Probably because they’re career hvac techs and it doesn’t occur to them that someone would want a freezer to stop working.
What were the hackers after? Ransom ware?
Seems like pure destruction to me, most likely from a nation state (who else would have motive, means, and opportunity?). Maybe it's retaliation for that Russian bot farm that got blown up?
Destruction and/or criminality. Next time someone in corporate HQ gets an email about "set up a crypto account to pay us $$$$ or else we shut your stores down", they'll know this isn't a spam email or pointless threat.
More options
Context Copy link
A practice run for the Chinese hackers who've already infiltrated much of American infrastructure.
MO seems more Russian than Chinese, Iranian is also technically possible. But I agree that hostile nation-state is the most likely motive.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
It's still weird that the guy building the web interface would do it, even if he was a career HVAC tech that had gotten pushed into software dev. But it's plausible.
That's a fun thought. Ransomware naively feels a little weird for me. Telling the people they need to pay you gives them a window to replace the controllers (or just turn off internet access and hope you didn't leave a time bomb). But I may be missing an attack vector there, or they might have been working toward the timebomb and just noticed enough devices they got interrupted.
There's some more esoteric options, though. Hitting food supplies is famously one a target for causing political unrest. Just shorting a few of the supermarkets would potentially be a very remunerative option, though it'd be a hilarious way for the hackers to get caught (or lose money). Burning Danfoss itself might be a thing, especially for the more ideological actors -- even if the threat genuinely doesn't exist for their newer products (not a given), there's no way its buyers avoid a security audit, and no way Danfoss avoids a ton of support costs downstream of that, on top of lost goodwill. Political or nationalist drives are always an option, if a goofy one. "Hey, Look At Me" or a personal grudge turned public is just plain dumb, but if any attack could be executed by a script kiddy, this one could.
Not sure some other options.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Seems to me that the smarter way to do this would be to have a site-to-site VPN between your corporate office and the store's internal network. Then the freezers remain on the "internal" corporate network and aren't exposed to the internet.
This may be shocking to professional software engineers, but most companies do not put systems on a VPN prior to those systems being hacked.
I guess a lot depends on when the system was put together, but as a professional I.T. guy, oof. Of course you put them on the VPN! That's been the sane choice for at least twenty years. If you absolutely, positively can't for some reason, then you at least whitelist the control server at the firewall level and block other traffic to the device.
Now if they were small enough not to have an I.T. department, I'd understand. But more likely I.T. was overruled.
It was probably put together by career HVAC techs who told them to set up communication via a web portal so they could initiate defrosts* on their cellphones, and then set the passwords to their lastnames.
*For those in the back who don't know, commercial freezers are supposed to have an electronic defrost that clears ice off the evaporator so it doesn't inhibit airflow. This attack worked by reprogramming that defrost to never end.
Most full-size residential fridge units have auto-defrost circuits. I'm not aware of having to run one manually before: it's probably not meant to be manually controlled like that. The little ones generally need to be taken outside to thaw occasionally. I believe heat pumps for HVAC need auto-defrost too.
Usually commercial freezers have a manual defrost switch to test functionality or restore operation after a freeze up caused by misuse, which is what they triggered.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Yes but that does require more infrastructure. It's also very possible that the units connect to a manufacturer cloud controller on the Internet only on outbound connections, which is typically reasonably secure so long as you use SSL pinning etc, or even on an IPSEC tunnel, but that the cloud controller was itself compromised in some way.
The manual for one of the affected models is available online, see page 11. The idea of a web host running on the hardware controller is... not the worst thing I've ever done, but :
Definitely worse. I was building web-accessible hardware in 2013. This was, to be fair, a couple years pre-Let's Encrypt, and SSL wouldn't have solved this specific problem.
But I still knew better than that.
Early descriptions of the attack have claimed that there was a master password for the device exposed on the same interface (unclear if it's the HTTP or XML), which allowed the control commands through in bulk.
Yeah, this sounds terrible. I wonder how hard it is to run Tailscale on a freezer. Surely the licensing costs for a freezer tailnet are miniscule compared to the reputational damage they have suffered.
A lot of this hardware predates tailscale, and pre-2016 VPNs were... very frustrating to manage even on normal desktop or laptop computers: you at least could run IPSec so the protocol situation wasn't awful, but the server management was either OpenVPN's too-techy-for-some-techs world, or expensive per-user licensed stuff, or untrustworthy stuff like Hamachi.
As for how difficult it would be to fix today... depends a lot on the hardware implementation. A pure MCU-driven approach would be cleaner, less expensive, and more stable, but also much harder to write a VPN or VPN-like capability into, just because of limited RAM and multithreading. An ESP32 can (and many people do) run it, but it's something you need to build into the system early and limits what else you can do. And 2010-era microprocessors would be really hard-pressed to run the stuff Danfoss's guides show, anyway.
It looks like they were on (Linux-based?) conventional embedded computers, at which point adding tailscale is mostly a matter of sysadmin work (and, uh, depending on how old the distro fork/variant, a lot of swearing about compile time and troubleshooting dependencies).
I mean, if this freezer can run an HTTP server with a control panel, it can certainly run Tailscale. I agree that older VPNs were terrible, I remember arranging them between us and our more paranoid customers (not the most paranoid ones, which demanded we keep maintaining our homebrew dial-up client) was a royal pain in the ass. Even bare Wireguard sucks.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
All of these manuals are online for technicians to read.
More options
Context Copy link
Oh my, that is indeed dire.
More options
Context Copy link
Geez, page 60 has a default admin password of 12345... There's a decent chance they just used that and then clicked the defrost button. It's fortunate that the attackers appear to have been unsophisticated, apparently you can also remotely upload new firmware and they don't describe any verification or signature validation.
Or maybe half a step more sophisticated they got VPN creds for the grocery store from a data breach or something, I doubt they were just sitting accessible on the internet. That's the sort of thing Shodan would have flagged years ago.
And right next to the default password:
More options
Context Copy link
Yes, all of that stuff is absolutely industry standard- because most of these freezers are worked on by contractors with an end user who don’t know nothing about nothing and they need a default admin password.
Their main competitor is even worse.
The default admin password would be common in normal IT world until 2020: it took a combination of California law (2018's SB-327) and NIST's mandate, along with some questionably-authorized-by-statute contracting pressures, that really made it universal for routers. I'll still see it (or password) on internal-only equipment, including some still-being-sold-as-new Unifi devices.
It's not even necessarily wrong in an environment where you'd expect some intrusion detection, or where the device requires a change on first login. It's just bizarre to do it, and then recommend people forward a WAN directly. This particular attack seems to have depending on another, different 'master' password, so I guess it doesn't even matter, but that's another thing that on its own points to 'reasonable if you're not that concerned about horizontal movement' (common in 2013) and really bad if you have any reason to worry about being the first hop in an infection.
Like, I'd hope they have fail2ban or equivalent set up? But I wouldn't be optimistic.
The competitor being worse makes me really concerned.
You are aware these are freezers, right? Like they're important but they're not holding onto private data or whatever.
Emerson writes the password on the device for technicians to have access to, and Ke2therm uses all fives(no, it can't be changed).
Writing the password on the device (in a place that isn't visible to shoppers with ordinary effort) seems obviously correct - you want a tech to be able to access the controls even if the manager who knows the password is out or has forgotten it. And protecting freezers from people who already have physical access to them is pointless.
More options
Context Copy link
The IT-world argument is that once you compromise any system on a subnet, you can at least listen to everything else on the subnet, and often attack everything else, too. While it's not entirely true in a modern world, it's pretty close. Conficker is the historical example of a wakeup call here. I get that HVAC techs don't really think that way (and, hell, some IT people don't, a la the 'air-gapped' networks that are really just bad firewall configs), though.
The HVAC argument is that a freezer failure is easily five digit cost of damage, even in the optimistic case where the food is just unsaleable rather than inedible, I can't imagine an expert working that job hasn't seen it. The defrost attack here might not be foreseeable for the average HVAC tech, who might reasonably expect it to 'just have a max timer', but in turn I don't think it's the only or even worst option. I don't have a good model of industrial freezers, but from what happens to consumer air conditioners or freezers when they're stuck short-cycling, I'm guessing that would be Bad, and the sort of failure that requires long technician hours and longer lead times on parts.
The former's actually pretty reasonable, at least if it isn't something stupid like 'password', and NIST recommends it for a lot of devices initial configuration. The latter, I'd like to just have a talk. Holy shit. That's literally worse than the 'some idiot has on his luggage' option.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
This system was not designed by IT professionals.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link