1. Stop putting mission critical systems on Windows, it's not the reliable OS it once was since MS has cut off most of its QA

2. AV solutions are unnecessary if you properly harden your system, AV was needed pre-Vista because Windows was literally running everything as Administrator. AV was never a necessity on UNIX, whatever MS bundles in is usually enough

3. Do not install third party software that runs in kernel mode. This is just a recipe for disaster, no matter how much auditing is done beforehand by the OEM. Linux has taught multiple times that drivers should be developed and included with the OS. Shipping random binaries that rely on a stable ABI may work for printers, not for mission critical software.

It's all about compensating controls.

It did involve a lot of documentation of inter-machine security controls, network access restriction and a penetration test by an offensive security company starting with a machine inside the network, but it can be done! Also in my opinion it gives you a more genuinely secure environment.

It just not as simple as commenters on this thread wish!

Yes you can go back and forth and argue the toss, but it pushes up the cost of the sale and forces your customer to navigate a significant amount of bureaucracy to get a contract agreed. Or you could just run AV like they asked you to...

If for instance they're remoting into a restricted VM all day, that's a different set of tradeoffs many might not be happy with.

The romans used to make the architects stand under the arches they built, to enforce the idea of consequences for bad work.

Corporate IT is always going to lean towards the "safe" compliance option.

What prevents someone pushing a malicious package that takes my user data (that is accessible from a logged in session directly) and sends it somewhere? Especially in non-system repos, like Maven/NuGet/npm/pip/RubyGems and so on? What about the too widespread practice of piping shell scripts from the web, or applications with custom update mechanisms that might be compromised and pull in malicious code?

I'm not saying that AV software would protect against all of these, but even if users don't do stupid things (which they absolutely will anyways, sooner or later), then there are still vectors of attack against any system.

As for why *nix systems don't see that much malware, I've no idea, maybe because it's not as juicy of a target because of the lower count of desktop software installations (though the stuff that is on the systems might be more interesting to some, given the more tech savvy userbase), or maybe because a lot of the exploits focus on server software, like most CVEs.

On Windows, I guess the built in AV software is okay, maybe with occasional additional scans by something like Malwarebytes, but that's situational.

https://arstechnica.com/information-technology/2021/12/malic...

AVs are the wrong way to go about security anyway, it's a reactionary strategy in a cat and mouse game by definition. For prevention, I think the BSDs are doing some promising work with the "pledge" mechanism. And as much hate as they get, I like appimages and snap et al for forcing people to consider a better segmentation model and permission system for installed software.

Its an extremely strict approach, but it does address the situation you're talking about.

endpoint security is a great utopia to strive for, but to get there we ought to starts with having secure by default endpoints.

In a perfect world, AV software wouldn’t be necessary. We don’t live in a perfect world. So we need defense-in-depth, covering prevention, mitigation, and remediation.

SELinux and whatever Apple is doing looks right IMHO

That's not an argument in good faith. If you install unvetted packages in your airline control system, bank, or supermarket, the kind of systems that we're talking about here, you have much bigger problems to worry about.

> I'm not saying that AV software would protect against all of these,

Or indeed any of these. Highly privileged users piping shell scripts from untrusted sources is out of scope for any antivirus system, on any platform.

That doesn't mean all platforms are identical, or share the same attack vectors. It is much more accepted to install kernel mode drivers on the Windows platform, where it is not only accepted but have established quality control programs to manage it, than on Linux, where the major vendor will very literally show you the middle finger on video for everyone to see for doing so.

The Linux community is more for doing that kind of work upstream. If some type of new access control or binary integrity checking is required, that work goes upstream for everyone to use. It is not bolted on running systems with kernel mode drivers. That is because Linux is more like a shared platform, and less like a "product". That culture goes way beyond mere technical differences between the systems.

Surely we can agree that if it's a vector with an above 0% chance of it being exploited, then any methods for mitigating that are a good thing. Quite possibly even multiple overlaid methods for addressing the same risks. Defense in depth and all, the same reason why many run a WAF in front of their applications even though someone could just say: "Just have apps that are always up to date with no known CVEs".

> Or indeed any of these. Highly privileged users piping shell scripts from untrusted sources is out of scope for any antivirus system, on any platform.

You don't even have to be highly privileged to steam information, e.g. an "app" for running some web service could still serve to exfiltrate data. As others have mentioned, maybe this is not what AV software has been historically known for, but definitely there are pieces of software that attempt to mitigate some of the risks like this.

I'd rather have every binary or piece of executable code be scanned against a frequently updated database of bad stuff, or use heuristics to figure out what is talking with what, or have other sane defaults like preventing execution of untrusted code or to limit what can talk to what networks, not all of which is always trivial to configure in the OSes directly (even though often possible).

I won't pretend that AV software is necessarily the right place for this kind of functionality, but I also won't pretend that it couldn't be an added benefit to the security of a system, while also presenting different risks and shortcomings (threat vector in of itself or something that impacts system stability at worst, or just a hog on the resources and performance in most cases).

Use separate VMs, use secret management solutions, use separate networks, use principle of least privilege, make use of good system architecture, have good OS configuration, use WAFs, use AV software, use scanning software, use dependency management alerting software, use static code analysis, use whatever you need to mitigate the risk of waking up and realizing that there's been a breach and that your systems are no longer your own.

Even all of that might not be enough (and sometimes will actually make things worse), but you can at least try.

The issue here is not simply scanning for known malware, "endpoint protection" systems go way beyond that. I have never, in practice, seen any of those systems be a net benefit for security. And I mean in a very serious and practical way. Depending on your needs, there are far more effective solutions that don't require backdooring your systems. There simply shouldn't be any unauthorized changes for this type of systems.

Agreed, most folks should probably use a proven *nix distro, or one of the BSD varieties. That would be a good starting point.

That said, I doubt whether the OS alone will be enough, even with a good configuration, but at some point the technical aspects have to contend with managing liability either way.

For 3 computers, 2 laptops, I've never _not_ had display bugs/oddities/issues. System upgrades always make me nervous because there is a very real chance of something getting fucked up and my screen staying black the next time it boots, having to go into a TTY, and manually fixing stuff up or booting the previous version that was still saved in GRUB.

We can not get computers perfect. They are too complicated. That's true for anything in life. As soon as it gets too complicated, you're left in a realm of statistics and emergent phenomena. As much as I dislike windows enough to keep using Linux, I never had display issues on windows.

To anyone compelled to reply with a text that contains "just" or "simply": simply just consider that if you are able to think of it in 10 seconds, then I have thought of it as well, and tried it too.

I've had a Debian update break GRUB itself as well: https://blog.kronis.dev/everything%20is%20broken/debian-and-...

I still use some Linux distros because when they work, they're pretty good, but when they don't, be prepared for a bunch of annoyances and debugging.

This is reasonable and sound.

> This stuff just happens _with desktop machines, on desktop OSes_

I wouldn't call Debian a desktop OS per se, though (albeit installing XFCE just in case does introduce a bit more risk of breakage).

Critique of consumer hardware is valid, but it's quite upsetting that bad software is the status quo.

Linux is used EVERYWHERE for a reason. Most car HUD now run on some form of Linux embedded, like basically all embedded and low power devices. The problem here is that people still put embedded mission critical systems on a desktop OS and slap desktop software on it, which is _a bad choice_.

You might say "that's not mission critical, you are just one guy and the company will keep running". I might say to literally anything "that's not mission critical, that is just one company and the world will keep turning".

In fact the companies with a sufficient cash reserve will likely just keep running anyway because this will be forgotten in a few months.

You may have had particularly bad luck with poorly supported hardware, but I don't think this is a normal experience.

I've been using Linux exclusively on desktops and laptops (with various VGA, DVI, DisplayPort, HDMI, and PD-powered DisplayPort-over-USB-C monitors and TVs since 2002 without any unstable behavior or incompatibility.

That's certainly a perspective.

This is demonstrably false, given the amount of people that game on Linux nowadays.

> System upgrades always make me nervous because there is a very real chance of something getting fucked up and my screen staying black the next time it boots, having to go into a TTY, and manually fixing stuff up or booting the previous version that was still saved in GRUB.

I had this happen to me once. Timeshift was painless to use, and in about 15 minutes I had my machine up and running again, and could apply all updates properly afterwards. If anything it made me bolder lol.

Every engineering-first or Internet-native company that could choose chose Linux and for simple reasons. Anything not Linux in The Cloud is a rounding error. Most of the world's mobile phones is Linux. And most cloud-first desktops too. They don't seem to be particularly more troubled with HDMI signal quality or other display issues than other devices.

It just just works for me, and has just worked with every laptop I have had in the last 15 years. My kids and I have several Linux installs and the only one with HDMI output issues is a cheap ARM tablet that is sold as a device for early adopters.

> For 3 computers, 2 laptops, I've never _not_ had display bugs/oddities/issues. System upgrades always make me nervous because there is a very real chance of something getting fucked up and my screen staying black the next time it boots, having to go into a TTY, and manually fixing stuff up or booting the previous version that was still saved in GRUB.

At least that number of machines (I do not know whether you mean three or five in total) for the last 20+ years and can recall one such issue.

But working with hundreds of computers, running many different distributions of Linux for decades, they just haven't ever seen what you're describing. It's really hard to reconcile what I read here with my hands-on experience.

Windows RTMs used to be shipped in a usable state (albeit buggy) for more than a decade. You installed it from a CD and it worked fine, you installed patches every once in a while from a random Service Pack CD you got from somewhere.

Modern Windows has had the habit of being so buggy after release in such horrendous ways that I can't imagine being able to use the same install CD for years. This is definitely putting less attention to detail in my view.

Yes they did. Hear about it first hand from an ex windows software tester:

https://www.youtube.com/watch?v=S9kn8_oztsA

https://www.youtube.com/watch?v=lRV6PXB6QLk

Take Mærsk who couldn't operate their freight terminals due to a cyber attack and had the entire operation being dependent on a hard drive in a server that happened to be offline. Have they improved network separation? Perhaps. Have they limited their critical infrastructure to only run whitelisted application? I assure you they have not. They've probably just purchased a Crowdstrike license.

Companies continuously fail to view their critical infrastructure as critical and severely underestimate risk.

[0] https://www.csoonline.com/article/567845/rebuilding-after-no...

Switching over to windows wouldn't just be extremely costly from an IT perspective but would require millions of dollars in new hardware. We are in the red in part because of the pandemic, existing problems in our industry accelerated by the last few years, and because a large percentage of our patients are on Medicare, which the fed govt shrinks fixed service payments for every year.

I can't imagine convincing our administration to switch over to Linux across the hospital without a clear, obvious, and more importantly short-term financial payoff.

Does this make financial sense? Probably not in the short run, which is an issue for most companies nowadays. But in the long run? I think it's the right choice.

Anyway, i totally agree with you. The convincing part here is short of clear and obvious for administration types. Until MS finally bricks it's OS and renders it totally unusable they can continue to do whatever shit they want and keep mocking their loyal customers forever.

True story.

1/ imagine running >1000 legacy applications, some never updated in 20 years 2/ imagine a byzantine mix of local data centers, VPCs in aws/gcp/azure 3/ imagine a IT departament run by a lot of people who have never learned anything new since they were hired

That would be your typical large, boring entity such as a bank, public utility or many of the big public companies.

Yeah, there is no law of physics preventing this, but it's actually nearly impossible to disentangle an organization from decades of mess.

People have continued to run old management systems inside of virtual machines and similar solutions. You can sandbox it, reset it, do all kinds of wondrous things if you use modern technologies in an era-appropriate way. Run your old Windows software inside of a VM, or tweak it to run well on Wine if you have the source. The reason this mess happened is that all of those software are literally running a desktop OS in mission critical applications.

I have worked as an embedded engineer for a while and I can't count the number of nonsensical stuff I've seen incompetent people running on unpatched, obsolescent Windows XP and 7 machines. This mess is 100% self inflicted.

Dream on, that ship sailed long, long, long ago.

It’s always funny to me reading comments on this site from users like this who have no idea how the real world operates

The definition of "operate" has changed over the years.

You mean don't install Steam nor the Epic Store, nor many of the games.

Note: I'm agreeing with you except that pretty much the only reason I have a Windows machine is for games. I do have Steam installed. I also have the Oculus software installed. I suspect both run in kernel mode. I have to cross my fingers that Valve and Facebook don't do bad things to me and don't leave too many holes.

I don't install games that require admin.

Oh, and I have Photoshop and I'm pretty sure Adobe effs with the system too >:(

To me, any app that asks for admin is suspect.

...would you install Steam on a POS machine? Is your gaming PC a "mission critical system"?

no software vendor needs to or wants to write a driver to spy on you or steal your data when they can do all of that with user-level permissions without triggering any AV.

3rd party drivers are completely fine, and its normal that advanced peripherals like an Oculus uses them

Have some kind of soaking/testing environment for production critical systems, especially if you're a big business. If you're hip, something like a proper blue/green setup (please chime in with best practices!). If you're legacy, do it all by hand if you must.

Blindly enabling immediate internet-delivered auto-update on production systems will always allow a bad update to cause chaos. It doesn't matter how well you permission things off on your favourite Linux flavor. If an update is to be meaningful, the update can break the software. And clearly you're relying on the software, otherwise you wouldn't be using it.

2. plenty of malware and c2 systems happily operate off all systems, regardless of how hardened (or how unix) they are - IDS/IPS is a reactive way to try and mitigate this

3. you don't need third party software to compromise the unix kernel, you just need to wait a week or two until someone finds a bug in the kernel itself

all that being said, this has solarwinds vibes. the push for these enterprise IDS systems needs to be weighted, the approach adjusted

It's a knee jerk reaction to the midwit zeitgeist. No real understanding of how the real world operates. No maturity or thoughtfulness.

Totally misleading prognosis. Misleading advice.

And when exactly was Windows a reliable OS ?

When it was turned off ? /s

Depending on the threats pertinent to the org they may require deep observability and the ability to perform threat hunting for new and emerging threats, or detect behaviour based signals, or move to block a new emerging threat. Not all threats require Administrator privileges!

Not installing AV might be fine for a small number of assets in a low risk industry, but is bad advice for a larger more complex environment.

If were unbiased here the apparent crowdstrike problem could occur on any OS and with any vendor where you have updates or configuration changes automatically deployed at scale.

I've had Linux GPU drivers fail multiple times due to system updates, to the point were I needed to roll back. I've had RHEL updates break systems in a way were even Red Hat support couldn't help me (I had to fix them myself).

I don't see how Linux is any better in this regard than Windows to be honest.

Also: > AV was never a necessity on UNIX

Sure, why write a virus when you can just deploy your malware via official supply chains?

I quite agree with OP here. VMs are now quite lightweight (compared to available resources on machines at least) and I would rather use a light, hardened Linux as my base OS that will run windows VM and do snapshots for quick rollbacks. Actually, that's what I run on my own PC, and I think it would be the sanest way to operate.

We don't have a good idea how to make software that is flawless, at least, not at scale for a cost that is acceptable. This is changing a little bit now with the drive by governments to use memory-safe languages, but that only covers a small part of the possible spectrum of bugs in software and hardware.

In this case it seems most of the software which is failing is dull back office stuff running on Windows - billing systems, train signage, baggage handling - which no one thought was critical, and there's no way on earth we could afford to rewrite it in the same way as we do aircraft systems.

Point of sale in a records store, less important. Point of sale in a pharmacy, could be problematic. Web shop customer call center, less important. Emergency services call center, could be problematic.

So if you develop your compression library it can't be used by anyone running critical infra unless you stamp it "critical certified", which in turn will make you liable for some quality issues with your software.

That's very realistic and already happens by requiring certain standards from the resulting product. For example, there are security standards and auditing requirements for medical systems, payment systems, cars, planes, etc.

Outside of regulated industries it's the context in which software is used which determines how critical it is. (As you say.)

So what you seem to be suggesting (effectively) is that use of software be regulated to a greater/lesser extent for all industries... and that just seems completely unworkable.

In physical world, you can specify a tolerance of 0.0005 in but the part is going to cost $25k a piece. It is trivially easy to specify tolerance, very hard to engineer a whole system that doesn't blow the cost and impossible to fund.

Great software architectures are the ones that operate cheaply, but are bulletproof when software fails. https://en.wikipedia.org/wiki/Chaos_engineering

Boeing disagrees.

We're talking about critical software. If we can't afford to reach the level of safety needed because it's too expensive, well so be it.

Besides, the enormously expensive flight systems don't seem to make my plane ticket expensive at all...

What failed today is a bunch of Windows stuff, of which there is a vast amount of software produced by huge numbers of companies, all of very variable quality and age.

Now, that it was not known previously to be critical, that may be. Whether we should have realised its criticality or not, is debatable. But going forward we should learn something from this. So maybe think more about cascading failures and classify more things as critical.

I have to wonder how the failure of billing and baggage handling has resulted in 911 being inoperative. I think maybe there's more to it than you mention here.

It's also common to rollout changes regionally to prevent global impact.

To me it seems Crowdstrike does not have a very good release process.

Here's something I don't understand: those jobs pay chump change compared to places like FB and (afaik) social networks don't have the same life-or-death context

That isn't an easy thing to do, but it should be possible.

There is no 'AI', that is always only hype. There is machine learning, which is a very powerful technology but I doubt MSFT will be leading that revolution. As for LLMs, MSFT might have some competitiveness there but I doubt it's going to be a very lucrative market. MSFT is highly overvalued.

I agree with you on the technical aspect, but the distinction makes regular people eyes glaze over within 5 seconds of that explanation. AI as a label for this is here to stay the same way cyber stopped meaning text sex of IRC. The people have spoken.

<< MSFT is highly overvalued.

Yes, but so is NVDA, the entire stock exchange and US real estate market. We are obviously due for a major correction and have been for a while. As in, I actually moved stuff around in my 401k to soften the blow in that event 2 years ago now. edit: yes, I am a little miffed I missed out on that ride.

So far, everything was done to prevent hard crash and in the election year, that is unlikely to change. Now after the election, that is another story altogether.

<< I doubt MSFT will be leading that revolution.

I think I agree. I remain mildly hopeful that the open model approach is the way.

You should stop trying to predict the next crash. According to the study, most people (including institutional investors) consistently believe there is a >10% chance the market will crash in the next 6 months when historically the probability is only 1%

Hmm? No. I will attempt to secure my own financial interest.

<< According to the study, most people (including institutional investors) consistently believe there is a >10% chance the market will crash in the next 6 months when historically the probability is only 1%

Historically is doing a fair amount of work here. I would argue there is little historical value to the data we face. Over the past few decades we went through through several mini revolutions ( industrial, information and whatever they end up calling now ) in terms of how we work, eat, communicate and, well, live.

All of these have upended how humans interact with the world effectively changing the calculus on the data that preceding it if not nullifying it altogether in some ways.

Your argument is to stop worrying since you are likely wrong anyway, by a factor of 10. I am saying is 1935 people also thought they have time to ride the wave.

edit: ok, need coffee. too many edits

Agree. First half of 2025 could be pretty spectacular (if/when we get through 2024).

I suspect there might be some pretty radical plans for US debt monetisation being drawn up, to be implemented early in the new presidential term.

Is that even possible any more? (That said, "operate" isn't a boolean, it's a continuum between perfect service and none, with various levels of degraded service between, even if you mean "operate" in the sense of "perform a surgical operation" rather than "any treatment or care of any kind").

All medical notes being printed in hard-copy could be done, that's the relatively easy part. But there's a lot of stuff which is inherently IT these days, gene sequencing, CT scans, etc., there's a lot that computers add which humans can't do ourselves — even video consultation (let alone remote surgery) with experts from a different hospital, which does involve a human, that human can't be everywhere at once: https://en.wikipedia.org/wiki/Telehealth

> Nothing critical / life-or-death / personal injury should rely on Windows / IT systems.

If you think that's bad, you may want to ensure you're seated before reading this about the UK nuclear deterrent: https://en.wikipedia.org/wiki/Submarine_Command_System

If someone is injured or dies because the hospital has inadequate backup processes in the event of a Windows outage, some or maybe all liability for negligence falls on those who designed the hospital that way, not the IT supplier who didn't agree to it.

The tool isn't fit for purpose

Many companies paying lip service to quality/reliability but internal incentives almost always go against maintenance and quality of service work (and instead reward new projects, features e. t. c.).

Of course it would. Restaurants are held liable for food poisoning, but they still operate just fine. They just - y’know - take care that they don’t poison their customers.

If computer systems were held liable, software would be a lot more expensive. There would be less of it. And it would also be better.

I think I can get behind that future.

Write me software that coordinates all flights to and from airports, capturing all edge-cases, that's bug free. Then tell me the number you estimate and the number of years to roll this out.

The right way to write code like that is to start simple and small - we're going to service airports X, Y and Z. Those airports handle Q planes per day. The software will be used by (this user group) and have (some set of responsibilities). The software engineers will work with the teams on the ground during and after deployment to make sure the software is fit for purpose. Someone will sign off on using it and trusting its decisions. And lets also do a risk assessment where we lay out all the ways defects in the software could cost money and lives, so we can figure out how risk averse we need to be.

Give me scope like that, and sure - I'll put a team together to write that code. It'll be expensive, but not impossible. And once its working well, I'd happily roll it out to more airports in a controlled and predictable manner.

Disclaimer. Neither Microsoft, nor the device manufacturer or installer, gives any other express warranties, guarantees, or conditions. Microsoft and the devicemanufacturer and installerexclude all implied warranties and conditions, including those of merchantability, fitness for a particular purpose, and non-infringement. If your local law does not allow the exclusion of implied warranties, then any implied warranties, guarantees, or conditions last only during the term of the limited warranty and are limited as much as your local law allows. If your local law requires a longer limited warranty term, despite this agreement, then that longer term will apply, but you can recover only the remedies this agreement allows.

So they might get sued on a local level?

Wouldn't that be the desirable outcome, though? Given the amount of damage they have caused, they should cease to exist.

We need this so that every company board is always asking "are we investing enough to make sure this never happens to us?"

Killing the company because they made a mistake doesn't just throw away a ton of learned lessons (because the devs will probably be scattered around the industry where their newly acquired domain knowledge will be less valuable) but also forces a lot of companies to spend resources changing their antivirus scanners. For all we know, Crowdstrike might never fuck up again after this and forcing that change would burn hundreds of millions for basically no reason.

I don't think that's right, since it ignores externalities.

You want to create a system where every company is incentivized to make positive security decisions. If your response to a fuckup of unprecedented scale is just "they learned their lesson, they probably won't do that again", then the message these companies receive is that it is okay to neglect proper security procedures, because you get one global economic meltdown for free.

It takes 2 year for the legal system to catch up, at which point he starts a new company, bankrupts the old one, sells all his tools cheaply to the new company, and fires and rehires his workers. I've seen this game going on for 14 years now.

I think Crowdstrike would do the same: Start a new one, sell the software, fire and rehire the workers, then go on as if nothing happened

Haven't heard that one before but I love everything about that!

https://www.scribbr.com/fallacies/appeal-to-emotion

Why not dilute the shareholder pool by a serious amount? There's no need for a statization to formally happen, the government can sell the shares back over time without actually exercising control.

Also fire execs and ban them from holding office on publicly traded companies for the foreseeable future.

Seizing shares doesn't impact the cash flow of the company directly, thus shouldn't cause job losses, but shareholders (who should put pressure on executives and the board to act with prudence to avoid these kinds of disasters) are adequately punished.

That could be amazing: "Ooopsie, in punishing Crowdstrike they've ended up folding and now there's a second global outage."

So if I own some Vanguard mutual fund as part of a retirement account, it’s now on me to put pressure on 500+ corporations?

Perhaps it’s on Vanguard to do so…but Vanguard isn’t going to just eat the cost of increased due diligence requirements. My fees will increase.

How does that increased due diligence even work? It’s not like I or Vanguard can see internal processes to verify that a company has adequate testing or backups or training to prevent cases like today’s failure.

When, on average, X number of those 500 companies in my mutual fund face this share seizure penalty per year…am I just supposed to eat the loss when those shares disappear? Does Vanguard start insuring against such losses? Who pays for that insurance in the end?

This doesn’t even really hurt the shareholders who are best placed to possibly pressure a company. This doesn’t hurt “billionaire executive who owns 40% of the outstanding shares”. I mean, sure, it will hurt that little part of their brain that keeps track of their monetary worth and just wants to see “huge number get huger”…but it doesn’t actually hurt them. It just hurts regular folks, as usual.

Consequently you don't put pressure on the 500 companies, you put pressure on the mutual fund and the mutual fund in turn puts pressure on the companies it invests in and exercises additional discretion in which companies it invests in.

>Perhaps it’s on Vanguard to do so…but Vanguard isn’t going to just eat the cost of increased due diligence requirements.

Yes they do, because mutual funds do compete with one another and a mutual fund that does the due diligence to avoid investing in companies that are held liable for these kinds of incidents will outperform the mutual funds that don't do this kind of due diligence.

> It’s not like I or Vanguard can see internal processes to verify that a company has adequate testing or backups or training to prevent cases like today’s failure.

I don't know specifically about Vanguard, but mutual funds in general do employ the services of firms like PwC, Deloitte, and KPMG to perform technical due diligence that assesses the target company's technology, product quality, development processes, and compliance with industry standards. VC firms like Sequoia Capital and Andressen Horowitz do their own technical due diligence.

You probably want, in addition to your proposal, executive stock-based compensation to be awarded in a different share class, used to finance penalties in such cases where the impact is deemed to be the result of gross negligence at the management level.

Do it to Boeing, sure.

That gives shareholders of other companies good reason to care going forward.

Same thing with AI. You can't punish an AI, it has no body.

But it is something to be very aware of for those of us who develop software run in e.g. hospitals and airlines, and should receive more attention, instead of only bringing up financial losses which is what usually happens. I noticed the same with the big ransomware attacks.

CrowdStrike knows that their software runs on computers that are in fricken hospitals and airports, they know that a mistake can potentially cause a human death. They also know how to properly test software, and they know how to do staggered releases.

Given what we know now, it seems pretty likely that to any reasonable person, the amount of risk they took when deploying changes to clients was in no way reasonable. People absolutely should go to jail for this.

This more or less originated with the unfortunately named MS Herald of Free Enterprise sinking (https://en.wikipedia.org/wiki/MS_Herald_of_Free_Enterprise) - after that incident, regulators decided that maybe they didn't want enterprise quite as free as all that, and cracked down significantly on shipping operators (though the attempt to prosecute its execs for corporate manslaughter did fail).

Why don't orgs test their updates? Every decent IT management/governance under the sun demands that you test your updates. How the hell did so many orgs that are ISO 2700x, COBIT, PCI-DSS, NIST CSF, etc. certified failed so hard??

(ToS/contracts will probably get you out of any damages.)

Because historically orgs have been really bad with applying updates: either no updates or delayed updates resulting in botnets taking over unpatched PC's. Microsoft's solution was to force the updates unconditionally upon everybody with very few opportunities to opt out (for large enterprise customers only).

Another complication comes from the fact that operating system updates are not essential for running a business and especially for small businesses – as long as the main business app runs, the business runs. And most businesses are too far removed from IT to even know what a update is and why it is important. Hence the dilemma of fully automated vs manually applied and tested updates.

Not a Microsoft's fan, but this is not true. Everyone who has Windows Server somewhere, with some spare disk space for the updates, has this ability. Just install and run WSUS (included in Windows Server) and you can accept/reject/hold indefinitely any update you want.

1) the prevailing majority of laptop and desktop PC installations (home, business and enterprise) are not Windows Server;

2) kiosk style installs (POS terminals, airport check-in stands etc) are fully managed, unsupervised installations (the ones that ground to a complete halt today) and do not offer any sort of user interaction by design;

3) most Windows Server installations are also unsupervised.

They are not, but the point is elsewhere: that Windows Server is going to provide the WSUS service to your network, so your laptop and desktop installations (in business and enterprise) are going to be handled by this.

Homes, on the other hand, do not have any Windows Server on their network, that's true.

As a hack to disable Windows updates, it is possible to point it to a non-existing WSUS server (so that can be done at home too). The client will then never receive any approval to update. It won't receive any info wrt available updates either.

> 2) kiosk style installs (POS terminals, airport check-in stands etc) are fully managed, unsupervised installations (the ones that ground to a complete halt today) and do not offer any sort of user interaction by design;

That's fine; this is fully-configurable via GPO.

> 3) most Windows Server installations are also unsupervised.

See 2.

What I take from this is that vendors need a LOT more investment in that work. They have both the money and are best positioned to do that testing since the incentives are aligned better for them than anyone else.

I’m also reminded of all of the nerd-rage over the years about Apple locking down kernel interfaces, or restricting FDE to their implementation, but it seems like anyone who wants to play at the system level needs a well-audited commitment to that level of rigorous testing. If the rumors of Crowdstrike blowing through their staging process are true, for example, that needs to be treated as seriously as browsers would treat a CA for failing to validate signing requests or storing the root keys on some developer’s workstation.

These are infra companies. Their incompetence can literally kill people.

So they drop half the 'safety' procedures once the auditor goes away? WTF! (I am semi-angry because there are so many easy solutions and workarounds to not fall for this!! (inside screaming).

How irresponsible must someone be to roll out something to 1k-5k-10k machines without testing it first??

Hubris-Atis-Nemesis-Tisis!!!!

https://www.greecehighdefinition.com/blog/hubris-atis-nemesi...

I'm not trying to enforce certifications because as a dev certifications always raise a bitter taste in my mouth. But those companies need certified processes that get re-certified every year. Sometimes even a cursory review from outsiders can find a lot of issues.

The reason this reminds me of that, assuming that I remember right, is that I think they had even taken the decision that the cost of paying lawsuits for those injuries was lower than the increase in revenue for being able to say "we have the hottest coffee"… and that was why they were deemed so severely liable.

They were definitely shown to have known it was resulting in injuries from other settlements:

https://en.wikipedia.org/wiki/Liebeck_v._McDonald%27s_Restau...

I'm not sure how i'm going to explain the productivity loss and retraining costs to the board if im honest.

You can switch away from CrowdStrike but I doubt you'll be able to convince whoever mandated CS to be installed to not install an alternative that carries exactly the same risks.

In fact there was a recent CrowdStrike-related crash in RHEL:

https://old.reddit.com/r/crowdstrike/comments/1cluxzz/crowds...

https://access.redhat.com/solutions/7068083

Crowdstrike only exists because Windows and other Microsoft products are so insecure their default configuration.

Every State Farm insurance office in the country is still using a DOS App from the 1980's to run their office.

These things should have gone from mainframes of yore to various unix systems, ideally a mix of different unix systems in hot failover.

Without running uncontrolled "agent" software of course.

Zero. Exactly Zero.

Clearly you have never been involved in buying insurance or writing contracts for IT products/services.

Loss of contracts, profits, goodwill, economic loss, loss of data and all that jazz is excluded in whole or limited to a fixed monetary value.

It is known as indirect, consequential or special loss, damage or liability.

No lawyer worth their salt will let an IT product/service company draft a contract that does not have the above type of clause..

And good luck finding an insurance contract that will pay out for such losses, indeed most of them have conditions that state your contracts with customers must exclude or limit such losses.

Most software also has clauses excluding use in safety critical environments.

If it was, it would also make it impractical for a small business to contract with a large one because of risk.

Enough of this limited liability nonsense, there need to be serious, severe, life-changing consequences.

Do these clients have SLAs? If so, they're definitely on the hook for something. You could probably get a few businesses together for a decent class-action against Crowdstrike. You're then expecting a lawyer to be able to convince a dozen semi-random people with varying degrees of computer knowledge that Crowdstrike's software was negligently designed, developed, and deployed in a way that caused financial or life losses for customers.

So, really, it's a coin flip.

The STEMI was stabilized, it's more that it was scary to lose every machine in the department at once while intubating a crashing patient. You're flying blind in a lot of ways.

If actually treating a patient with life saving measures during a heart attack isn't "much of anything" I don't know what is.

So, yeah. The paperwork can save lives too and not sadly red tape is bad.

Otherwise you may go o hospital to pickup your friend and be told to wait for coroner

Wouldn't that be nice

Sounds like with the systems being down the call would have been cut off which sounds horrible.

I feel your pain. Perhaps it is time to increase the karma level required for downvoting.

Regardless, thanks for your report; seeing it was very sobering. I hope you can get some rest, and that things will soon return to normalcy.

We were back in the 1960's with paper and pen for everything, no updates on nature of call, no address information, nothing... find out when you show up and hope the scene is secure. It was wild as it was coupled to a relatively intense Monsoon storm.

https://news.ycombinator.com/newsguidelines.html

https://news.ycombinator.com/item?id=41005274

I've told my testers for years their efficacy at their jobs would be measured in unnecessary deaths prevented. Nothing less. Exactly this outcome was something I've made unequivocally clear was possible, and came bundled with a cost in lives. Yet the "Management and bean counter types" insist "Oh, nope. Only the greenbacks matter. It's the only measure."

Bull. Shit. If we weren't so obsessed with imaginary value attached to little green strips of paper, maybe we'd have the systems we need so things like this wouldn't happen. You may not be able to enumerate every, but you damn well can enumerate enough. Y'all just don't want to because then work starts looking like work.

That doesn’t count serious bodily injury, suffering, people who were victimized, people who had their lives set back for decades due to a missed opportunity, a person who missed the last chance to visit a loved one, etc.

There are uncountable different impacts that happen when you’re talking about events on the scale of an economy. Which is why economists use dollars. The proxy isn’t useful because it is more important than life, it it useful because the diversity of human experience is innumerable.

At least putting a number to life is an genuine attempt even though it may be distasteful.

The fact is that there already is a number on it, which one can derive entirely descriptively without making moral judgements. Insurance companies and government social security offices already attempt to determine the number.

The number is not infinite or we'd have no cars.

Effecting airlines/brodcasters and banks.

https://www.nytimes.com/live/2024/07/19/business/global-tech...

There's a very good reason airport traffic control exists.

The issue wasn't with landing specifically; I'm just using it as a general example of issues caused by havoc situations in aviation.

The same cannot be said for medical records.

If you rely on your applications to be available you should have disaster recovery plans for scenarios like this.

Was this problem caused by Microsoft? No.

Why does this tool exist and must be installed on servers? Well, Windows OS design definitely plays a role here.

Why does this software run in a critical path that can cause the machine to BSOD? This is where the OS is a problem. If it is fragile enough that a bad service like this can cause it to crash in an unfixable state (without manual intervention), that’s on Windows.

Fads, laziness, and lack of forethought. This tool didn't exist a few years ago. Nobody stopped IT departments worldwide and said "hey, maybe you shouldn't be auto-rolling critical software updates without testing, let alone doing this via a third-party tool with dubious checks."

This could have happened on any OS. Auto deployment is the root problem.

So the only blame Windows can take is its widespread usage, compared to Debian.

so windows can still be blamed for not providing a relatively safe way of doing this.

Kernel panic observed after booting 5.14.0-427.13.1.el9_4.x86_64 by falcon-sensor process.

eBPF program causes kernel panic on kernels 5.14.0-410+ .

Apparently not safe enough for CrowdStrike.

Yes, the Linux device driver has many of the same issues (monolithic drivers running in kernel space/memory). I’m not sure what the mitigations were in that case, but I’d be interested to know.

But we both know this isn’t the only model (and have commented as such in the thread). MacOS has been moving away from this risk for years, largely to the annoyance of these enterprise security companies. The vendor that was used by an old employer blamed Apple on their own inability to migrate their buggy EDM program to the new version of macOS. So much so that our company refused to upgrade for over 6 months and then it was begrudgingly allowed.

This really should be a hell no. Perhaps Microsoft's greatest claim to fame is their enduring ability to quickly and decisively react to security breaches. Their process is extremely public and hasn't significantly changed in decades.

If your company can't work with Microsoft's process, your company is the problem. Every other software company in the last forty years has figured it out.

Had a few calls with them to figure out its features and how it would impact the systems. They didn't know.

I imagined a lot of this ran on some custom or more obscure and hardened specialty system. One that would generally negate the need for antiviruses and such. (and obviously, no, not off the shelf Linux/BSD either)

For something truly mission critical, I’d expect something more bespoke with smaller complexity surface. Otherwise Android is actually not a bad choice.

A lot of the new forced updates on Android is to prevent people some apps from being used to spy on other apps, stealing passwords, notification backdoor etc, but you don't need that if it's just a car radio.

If the alternatives/plan b's were as good or better than the plan a's then they wouldn't be the alternatives. Nobody is going to have half a hospital's care capacity sit as backup when they could use that year round to better treat patients all the time, they just have plans of last resort to use when what they'd like to use isn't working.

(worked healthcare IT infrastructure for a decade)

I worked for a company that sold and managed medical radiology imaging systems. One of our customers' admins called and said "Hey, new scans aren't being properly processed so radiologists can't bring them up in the viewer". I told him I'd take a look at it right away.

A few minutes later, he called back; one of their ERs had a patient dying of a gunshot wound and the surgeon needed to get the xray up so he could see where the bullet was lodged before the guy bled out on the table.

Long outages are terrifying, but it only takes a few minutes for someone to die because people didn't have the information they needed to make the right calls.

You don't need 4 years of specialized training to see a bullet on a scan.

X-Ray has limitations though - most of our emergencies aren't as easy to diagnose as bullets or pneumonia. CT, CTA, and to a lesser extent MRI are really critical in the emergency department, and you definitely need four years of training to interpret them, and a computer to let you view the scan layer-by-layer. For many smaller hospitals they may not have radiology on-site and instead use a remote radiology service that handles multiple hospitals. It's hard to get doctors who want to live near or commute to more rural hospitals, so easier for a radiologist to remotely support several.

One day, during a rapid pediatric patient intervention, a caregiver tried to log in to a PC to check a drug interaction. The computer took a long time to log in because of a VDI problem where someone had stored many images in a file that had to be copied on login. While the care team was waiting for the computer, an urgent decision was made to give the drug. But a drug interaction happened — one that would have been caught, had the VDI session initialized more quickly.

The patient died and the person whose VDI profile contained the images in the bad directory committed suicide. Two lives lost because files were in the wrong directory.

Seems like a possible plan would be duplicate computer systems that are using last week's backup and not set to auto-update. Doesn't cover you if the databases and servers go down (unless you can have spares of those too), but if there is a bad update, a crypto-locker, or just a normal IT failure each department can switch to some backups and switch to a slightly stale computer instead of very stale paper.

The "downtime" computers were affected just like everything else because there was no network.

Phones are all IP-based now; they didn't work.

Couldn't check patient histories, couldn't review labs, etc. We could still get drugs, thankfully, since each dispensing machine can operate offline.

Money spent on spares is not spent on cares.

Or maybe I'm giving these institutions too much credit?

You come in for you next shift and are finishing charting from your prior shift. You open one of your partially finished charts and a little popup tells you "you are editing the chart for a deceased patient".

This is why I'm impressed by anyone who works in a hospital, especially the more urgent/intensive care

We don't ask customers to switch all systems from Windows to Ubuntu, but to consider moving maybe a third to Ubuntu so they won't sit completely helpless next time Windows fail spectacularly.

While I see more and more Ubuntu systems, and recently have even spotted Landscape in the wild I don't think they were as successful as they hoped with that strategy.

That said, maybe there is a silver lining on todays clouds both WRT Ubuntu and Linux in general, and also WRT IT departments stopping to reconsider some security best practices.

Honestly your comment highlights one of the few defenses... don't sit all on one platform.

The specific of this CrowdStrike kernel driver (which AFAIK is intended to intercept and log/deny syscalls depending on threat assessment?) means that this is badnewsbears no matter which platform you're on.

Like sure, if an OS is vulnerable to kernel panics from code in userland, that's on the OS vendor, but this level of danger is intrinsic to kernel drivers!

Not with an immutable OS, which would make a rollback trivial (reboot, select previous configuration). I hope organizations learn from this, and we move to that model for all major OSes.

Immutability is great, as we know from functional programming. Nix and Guix are pushing this ideas forward, but there's no reason other OSes can't have them as well.

CloudStrike only uses a kernel level driver on Windows. It's not necessary for Mac, it's not necessary for Linux.

Why did they feel that they needed kernel level interventions on Windows devices specifically? Windows may have some blame there.

I do think there's an argument to be made that CrowdStrike is more invasive on Windows because Windows is intrinsically less secure. If this is true then yeah, MSFT has blame to share here.

I imagine they've simply moved to eBPF if they're not shipping the kernel module anymore.

In particular, while it looks like macOS's Endpoint Security API[0] and Linux 4.x's inclusion of eBPF are both reasonably robust (if the literature I'm skimming is to be believed), ETW is still pretty susceptible to blinding attacks.

(But what about PatchGuard? Well, as it turns out, that doesn't seem to keep someone from loading their own driver and monkey patching whatever WMI_LOGGER_CONTEXT structures they can find in order to call ControlTraceW() with ControlCode = EVENT_TRACE_CONTROL_STOP against them.)

0: https://developer.apple.com/documentation/endpointsecurity

Maybe because everyone else in "security" and DRM does it, so they figured this is how it's done and they should do it too?

My prior on competence of "cybersecurity" companies is very, very low.

What DRM uses kernel drivers? And how do you plan to prevent malware from usermode?

Dmitri Alperovitch agrees with you.[0] He went on record a few months back in a podcast, and said that some of the most atrocious code he has ever seen was in security products.

I am certain he was implicitly referring, at least in part, to some of the code seen inside his past company's own code base.

0: https://nationalsecurity.gmu.edu/dmitri-alperovitch/ ["Co-founder and former CTO of Crowdstrike"]

Crowdstrike uses a kernel level driver ONLY on Windows.

Even better..

ONLY on Windows does CrowdStrike use a kernel level driver.

On Linux, ebpf provides an alternative, and I assume, plenty of advantages over trying to maintain kernel level extensions.

I haven’t researched, but my guess is that Microsoft hasn’t produced a suitable alternative for Windows security vendors.

Windows offers it's users nothing here.

Of course that means putting the user in control of when they apply updates, but maybe that would be a good thing anyway.

[0] https://en.m.wikipedia.org/wiki/Gros_Michel_banana

Debian has automatic updates but they can be manual as well. That's not the case in Windows.

The best practice for security critical infrastructure in which peoples lives are at stake, is to install some version of BSD stripped down to it's bare minimum. But then the company has to pay for much more expensive admins. Windows admins are much cheaper and plentiful.

Also as a user of Ubuntu and Debian for more than a decade, i have a hunch that this will not happen in India [1].

[1] https://news.itsfoss.com/indian-govt-linux-windows/

Or they never considered it, which is far worse.

I haven't used ubuntu desktop in 12 years because of that nonsense. I'm forced to use ubuntu server, though - although thanks to python3 venv and docker that's becoming less so as time wears on.

Ubuntu spent a lot of time, talent, and treasure on trying to migrate people off windows instead of being a consistent, great OS. It is still with great dread that i open docs for some new package/program linked to from HN or elsewhere; dread that the first instruction is "start with ubuntu 18.04|20.04".

[0] They actually maintained the unofficial gentoo AWS images for over a decade. unsure if they still do, it could be automated to run a new build off every quarter. https://github.com/genewitch/gentoo/blob/master/gentoo_auto.... (a really old version of the script i keep to remind me that automation is possible with nearly everything...)

I'm just saying what they said their strategy was, not judging their sales people.

Yes, distribute your eggs, but check the handles on the baskets being sold to you by the guy pointing out bad handles.

Stable Ubuntu core under the surface, and everything desktop related delivered by the KDE team.

With that said, Microsoft could've done this with Defender just as easily, so be mindful of system diversity in your business continuity and disaster recovery plans and enterprise architecture. Heterogeneous systems can have inherent benefits.

https://apnews.com/article/tech-outage-crowdstrike-microsoft...

> “This is a function of the very homogenous technology that goes into the backbone of all of our IT infrastructure,” said Gregory Falco, an assistant professor of engineering at Cornell University. “What really causes this mess is that we rely on very few companies, and everybody uses the same folks, so everyone goes down at the same time.”

The irony is the NHS likely installed CrowdStrike as a direct reaction to WannaCry.

At large-scale, you don’t solve problems, you only replace them with smaller ones.

If you want actually good payout, your crypto locker has to either encrypt network filesystems, or infect crucial core systems (domain controllers, database servers, the filers directly, etc).

Ransomware getting smarter about sideways movement, and proper data exfiltration etc attacks, are part of what led to proliferation of requirements for EDRs like Crowdstrike, btw

But that's besides the point. Point is, attacks distributed over time and space ultimately make the overall system more resilient; an attack happening everywhere at once is what kills complex systems.

> Ransomware getting smarter about sideways movement, and proper data exfiltration etc attacks, are part of what led to proliferation of requirements for EDRs like Crowdstrike, btw

To use medical analogy, this is saying that the pathogens got smarter at moving around, the immune system got put on a hair trigger, leading to a cytokine storm caused by random chance, almost killing the patient. Well, hopefully our global infrastructure won't die. The ultimate problem here isn't pathogens (ransomware), but the oversensitive immune system (EDRs).

These security software vendors have found a wonderful tacit moat: they have managed to infect various questionnaire templates by being present in a short list of "pre-vetted and known" choices in a dropdown/radiobutton menu. If you select the sane option ("other"), you get to explain to technically inept bean counters why you did so.

Repeat that for every single regulator, client auditing team, insurance company, etc. ... and soon enough someone will decide it's easier and cheaper to pick an option that gets you through the blind-leading-the-blind question karaoke with less headaches.

Remember: vast majority of so-called security products are sold to people high up in the management chain, but they are inflicted upon their victims. The incentives are perverse, and the outcomes accordingly predictable.

Tell them it’s for preserving diversity in the field.

For anyone browsing the thread archive in the future: you can have that quip in your back pocket and use it verbally when having to discuss the bingo sheet results with someone competent. It's a good bit of extra material, but it can not[ß] be your sole reason. The term you do want to remember is "additional benefit".

The reasons you actually write down boil down to four things. High-level technical overview of your chosen solution. Threat model. Outcomes. And compensating controls. (As cringy as that sounds.)

If you can demonstrate that you UNDERSTAND the underlying problem, and consider each bingo sheet entry an attempt at tackling a symptom, you will be on firmer ground. Focusing on threat model and the desired outcomes helps to answer the question, "what exactly are you trying to protect yourself from, and why?"

ß: I face off with auditors and non-technical security people all the time. I used to face off with regulators in the past. In my experience, both groups respond to outcome-based risk modeling. But you have to be deeply technical to be able to dissect and explain their own questions back to them in terms that map to reality and the underlying technical details.

This should be the standard for any life sustaining or surgical systems, and any critical weapons systems.

Most of what I do is creating the tools to let the field reps go into hospitals and update capital equipment in a disconnected state (IE, the reps must be physically tethered to the device to interact with it). The fact that any critical equipment would get an auto-update, especially mid-surgery is incredibly bad practice.

Anyway: it is getting to the point that I cynically predict we may be required to add things to the system (such as embedding PCs), just so we can turn around and "secure" them to comply with the requirements that shouldn't be applied to these systems. Maybe this current outage event will be a wake up call to how misplaced the priorities are, but I doubt it.

Has to be wifi because the carts the nurses use roll around. Has to be networked so you can have EMR's that keep track of what your patients have gotten and the Pharmacists, doctors, and nurses can interface with the Pyxis machines correctly. The nurse scans a patients barcode at the Pyxis, the drawer opens to give them the drugs, and then they go into the patient's room and scan the drug barcode and the patients barcode before administering the drug. This system is to prevent the wrong drug from being administered, and has dramatically dropped the rates of mis-administering drugs. The network has to be everywhere on campus (often times across many buildings). Then the doctor needs to see the results of the tests and imaging- who is running around delivering all of these scans to the right doctors?

You don't know what you are talking about if you think this is easy.

If the system has to be networked with the outside world, who is responsible for physically updating all of these machines, so they don't get ransomware'd? Who has to go out and visit each individual machine and update it each month so the MRI machine doesn't get bricked by some teen ransomware gang? Remember that was the main threat hospitals faced 3-4 years ago, which is why Crowdstrike ended up on everyone's computer: because the ransomware insurance people forced them too.

There is a reason that I am a software engineer and not an IT person. I prefer solving more tractable problems, and I think proving p!=np would be easier than effectively protecting a large IT network for people who are not computing professionals.

One of my favorite examples: in October 2013 casino/media magnate and right wing billionaire Sheldon Adelson gave a speech about how the US and Israel should use nuclear weapons to stop Iran nuclear program. In February 2014 a 150 line VB macro was installed on the Sands casino network that replicated and deleted all HDDs, causing 150 million dollars of damage. That was to a casino, which spends a lot of money on computer security, and even employs some guys named Vito with tire irons. And it wasn't nearly enough.

The manufacturer does. As I mentioned in my OP I help build the software for our field reps to go into hospitals and clinics to update our devices in a disconnected state. Most of the critical equipment we manufacture has this as a requirement since it can't be connected to a network for security reasons.

As for discharge orders, etc, I can't speak to that, but that's also not what I would consider critical. I'm talking about things like surgical robots, which can not be connected to a network for obvious reasons, especially during a surgery.

> Adama: It's an integrated computer network, and I will not have it aboard this ship.

> Roslin: I heard you're one of those people. You're actually afraid of computers.

> Adama: No, there are many computers on this ship. But they're not networked.

> Roslin: A computerized network would simply make it faster and easier for the teachers to be able to teach--

> Adama: Let me explain something to you. Many good men and women lost their lives aboard this ship because someone wanted a faster computer to make life easier. I'm sorry that I'm inconveniencing you or the teachers, but I will not allow a networked computerized system to be placed on this ship while I'm in command. Is that clear?

> Roslin: Yes, sir.

> Adama: Thank you. 'Scuse me.

... at which point you will lose battles to enemies who have successfully networked their command and control operations. (For extra laughs, just wait until this is also true of AI.)

Ultimately there are just too darned many advantages to connecting, automating, and eventually 'autonomizing' everything in sight. It sucks when things don't go right, or when a single point of failure causes a black-swan event like this one, but in an environment where you're competing against either time or external adversaries, the alternatives are all worse.

A better thing to do is do phased deployment, so you can see if an update will cause issues in your environment before pushing it to all systems. As this incident shows, you can’t trust a software vendor to have done that themselves.

I've mostly resigned myself today to deploying the configuration change and watching for anomalies in my monitoring for a number of hours or days afterward, but I acknowledge that I also have both a process supervisor that will happily let me crash loop my programs and deployment infrastructure that will nonetheless allow me to roll things back. Without either of those, I'm honestly at a loss as to how I'd safely operate this product.

  # Update A
  
  ## config.ext
  
  foo = false
  
  ## src.py
  
  from config import config
  
  if config('foo'):
      work(2 / 0)
  else:
      work(10 / 5)

  # Update B
  
  ## config.ext
  
  foo = true

The most insidious part of this is when there are entire swaths of infrastructure in place that circumvent the usual code review process in order to execute those configuration changes. Boolean flags like your `config('foo')` here are most common, but I've also seen nested dictionaries shoved through this way.

In theory you could build an air-gapped network within a hospital, but then how do you transmit updates to the EMR's across different campuses of your hospital? How do you issue electronic prescriptions for patients to pick up at their home pharmacy? How do you handle off-site data backup?

Quite honestly, outside of defense applications I'm not aware of people building large air-gapped networks (and from experience, most defense networks aren't truly air-gapped any more, though I won't go into detail). Hospitals, power plants, dams, etc. all of them rely heavily on computers these days, and connect those over the regular internet.

1: My wife was the only pharmacist in her department last night whose computer was unaffected by Crowdstrike (for unknown reasons). She couldn't record her work in the normal ways, because the servers were Crowdstrike'd as well. So she spun up a document of her decisions and approvals, for later entry into the systems. It was over 70 pages long when she went off shift this morning. She's asleep right now.

2: https://www.bd.com/en-uk/products-and-solutions/products/pro...

TIP: many buildings can be part of one LAN! It is called VPN and Russia and China do not like it becouse it is good for peoples!

TIP: data can be easily exchanged when needed! Including LAN.

--

My wife is a hospital pharmacist. (1) When she gets a new prescription in, she needs to see the patients charts on the electronic medical records, and then if she approves the medication a drawer in the Pyxis cabinet (2) will open up when a nurse scans the patients barcode, allowing them to remove the medication, and then the nurse will scan the patient's barcode and the medication barcode in the patients room to record that it was delivered at a certain time. Computers are everywhere in healthcare, because they need records and computers are great at record-keeping. All of those need networks to connect them, mostly on wifi (so the nurses scanners can read things).

--

It was description of very local workflow...

It was description of data flow - no any reason it should be monopolized by unsecure by design os vendor that need to be mandatory secured by essentialy kernel rootkit aka os hacking. Which contradicts using that os in the first place!

And looks like Crowdstrike is just if you ask for price then you can't have it version of SELinux :>>> RH++ for two decades of making presentations of SELinux necessity.

But over all allowing automatic updates from 3rd party not having clue about medicine to hospital system, etc. is managers criminal negligence. Simple as that. Curent state of the art ? More negligence! Add (business) academia & co to chronic offenders. Call them what they truly are - sociopaths via craft training facilities.

>In theory you could build an air-gapped network within a hospital, but then how >do you transmit updates to the EMR's across different campuses of your hospital?

How do you transmit to other campuses of other hospitals ? EASY! Transfer mandatory data. Pleas notice I used words like "mandatory" and "data". I DID NOT SAY "use mandatory http stack to transfer data"! NO. NO, I'm far, faaar from even sugesting THAT ! :>

>How do you issue electronic prescriptions for patients to pick up at their home pharmacy?

Hard sold on that "air-gapped and in cage" meme, eh? Send them required data via secure and private method! Communications channels already "hacked" - monopolized - by FB? Obviously that should do not happend in first place. So resolve it as part of un-win-dosing critical civilian infra.

>How do you handle off-site data backup?

That one I do not get. You saying that cloud access is a only possibility to have backups??? And Internet is a must to do it?? Is medical staff brain dead? Ah, no... It's just managers... Again.

>Quite honestly, outside of defense applications I'm not aware of people building large air-gapped networks

And dhcp and "super glue" and tons of other things was invented by military, for a reason, but that things proliferated to civilians anyway. For good reasons. Air-gapping should be much more common when wifi signal allows tracking how you move in your own home. Not to mention GSM+ based "technologies"...

There is old saying: Computers maximize doing. And when somewhere is chaos then computers simply do their work.

If we adjusted our foreign policy slightly, I think we would dissuade that whole class of attacker.

Anecdotally, I have zero stability problems on my non-ECC consumer-grade 11th gen Intel Windows 11 system. It'll stay up for months, until I decide to shut it down. I had a loose GPU power cable that was causing me problems at a point, but since I reseated everything I haven't had a single issue. That was my fault, things happen. The system is great.

More significantly, I see no difference in stability between our Windows Server platform and Red Hat Enterprise (Oracle) server platform at work either. Work being one of the top 3 largest city governments in the USA.

didnt red hat have a massive DEI/anti white man scandal? I wouldnt trust their products

the smartest people use and maintain Arch, ergo everything should run on Arch for maximum stability

The problem is the middle management class at hospitals, governments, etc., only know how to use Word and maybe Excel, so they are comfortable with Microsoft, even though it's objectively the worst option if you aren't gaming. So then they make contracts with Microsoft and all the computers run Windows, so all the app developers have to write the apps for Windows.

lol no

For instance, if you had to make deploy a mission critical server, assuming cost and other software was the same, would you choose Linux or Windows for reliability? Of course you would choose Linux.

But Linux isn't immune from this exact sort of issue, though - these overgrown antivirus solutions run as kernel drivers in linux as well, and I have seen them cause kernel panics.

Depends i think. When i was working as a super market cashier the tils had embedded XP. in 2 or 3 years it rarely had issues. The rare issues it did have were with the java POS running on top.

Windows 10 for my home desktop crashed a lot more and just seems to have gotten more "janky" with time.

I used to work at MS and didn’t like their 2:1 test to dev ratio or their 0:1 ratio either and wish they spent more work on verification and improved processes instead of relying on testing - especially their current test in production approach. They got sloppy and this was just a matter of time. And god I hate their forced updates, it’s a huge hole in the threat model, basically letting in children who like to play with matches.

My important stuff is basically air-gapped. There is a gateway but it’ll only accept incoming secure sockets with a pinned certificate and only a predefined in-house protocol in that socket. No other traffic allowed. The thing is designed to gracefully degrade with the idea that it’ll keep working unattended for decades, the software should basically forever so long as equivalent hardware could be found.

My windows machine notified me of the update, asked me to restart. I was busy, so I didn't. Then the news broke, then the update was rolled back.

An OS should be really resistant to this kind of things.

That would have prevented a bad driver from taking down a device.

The system volume is signed by Apple. If the signature on boot doesn't match, it won't boot.

When the system is booted, it's in read-only mode, no way to write anything to it.

If you bork it, you can simply reinstall macOS in place, without any data/application loss at all.

Of course, if you're a tinkerer, you can disable both, the SIP, and the signature validation, but that cannot be done from user-space. You'll need to boot into recovery mode to achieve that.

I don't think there's anything in NTFS or REFS that would allow for this approach. Especially when you account for the wide variety of setups on which an NTFS partition might sit on. With MBR, you're just SOL instantly.

Apple hardware on the other hand has been EFI (GPT) only for at least 15 years.

If something cannot be formally verified at the machine code level there should be a controls level verification where vendors demonstrate they have a process in place to achieving correctness by construction.

Driver devs can be quite sloppy and copy paste bad code from the internet, in the machine code Microsoft can detect specific instances of known copy and pasted code and knows how to patch it. I know they did this for at least one common error. But if I was in the business of delivering an OS I want people to rely on my OS this stuff formal verification at some level would be table stakes.

2) it is not disabling them; it is approving or rejecting them (or even holding up the decision indefinitely).

You can do that too, via WSUS. It is not reserved to large enterprises, as I've seen claimed several times in this thread. It is available to anyone, who has Windows Server in their network and is willing to install the WSUS role here.

It's missing because users are not to be trusted with such things, and that's a philosophy with harmful consequences.

https://support.microsoft.com/en-us/windows/advanced-startup...

> Last Known Good Configuration (advanced). Starts Windows with the last registry and driver configuration that worked successfully.

And I don't think that's an accident either. Microsoft is not interested in providing end users with the kind of rollback functionality that you see in Linux (you can just pick which kernel to boot to) because you can get less money by empowering your users and more money by cooperating with people who want to spy on them.

The power to intrude on our systems and run arbitrary code aggregates in the hands of people that we don't know unless we're clever enough to intervene. That's not something to be celebrated. It's creepy and we should be looking for a better way.

We should be looking for something involving explicit trust which, when revoked at a given timestamp, undoes the actions of the newly-distrusted party following that timestamp, even if that party is Microsoft or cloudstrike or your sysadmin.

Sure, maybe the "sysadmin" is good natured Chuck on the other side of the cube partition: somebody that you can hit with a nerf dart. But maybe they're a hacker on the other side of the planet and they've just locked your whole country out of their autonomous tractors. No way to be sure, so let's just not engage in that model for control in the first place. Lets make things that respect their users.

It never caught any legit threat, but constantly flagged our own code. Our devs talked to security every other week to explain why this new line of code is not a threat. It generated a lot of work and security team's headcount just exploded. The software checked a lot of security checkboxes, and our CISO can sleep better at night, so I guess end of day it's all worth it.

When I worked in large enterprise it got to the point that if a piece of my app infrastructure started acting weird the blackbox security agents on the machines were the first thing I suspected. Can't tell you how many times they've blocked legit traffic or blown up a host by failing to install an update or logging it to death. Best part is when I would reach out to the teams responsible for the agents they would always blame us, saying we didn't update, or weren't managing logs etc. Mind you these agents were not installed or managed by us in any way, were supposed to auto update, and nothing else on the system outran the logrotate utility. Large enterprise IT security is all about checking boxes and generating paperwork and jobs. Most of the people I've interacted with on it have never even logged into a system or cloud console. By the end I took to openly calling them the compliance team instead of the security team.

I day-dream about being able to use immutable unikernels running on hypervisors so that even if something was to get past a gateway there would be no way to modify the system to work in a way that was not intended.

Air-gapping with a super locked down gateway was already getting more popular precisely due to the forced updates threat surface area, and after today I expect it to be even more popular. At the very least I’ll be able to point to this instance when explaining the rational behind the architecture which could help in getting exemptions from the antivirus box ticking exercise.

There is a windows release preview channel that exists for finding issues like this ahead of time.

To be fair - it is possible the conflicting OS update did not make it to that channel. It is also possible it is due to an embarassing bug from MSFT (uknown as yet).

Until I hear that this is the case - I am pinning this on Crowdstrike. This should have been caught before prod.

There is the correctness by testing vs correctness by construction dynamic and in my view given the scale of interactions between an OS and the kernel modules trying to achieve correctness by testing is negligent. Even at the market scale Microsoft has there are not enough Windows computers to preview test every combination. Especially when taking into account the people on the preview ring have different behaviors to those on the mainline so many combinations simply won't appear in the preview.

I see it as Microsoft owning the Windows kernel module space and has allowed sloppiness by third parties and themselves, I don't know the specifics but I could easily believe that this is a due to a bug from Microsoft. The problem with allowing such sloppiness is that the slopy operators out compete the responsible operators, the bad pushes out the good until only the bad remains. A sloppy developer can push more code and gets promoted while the careful developer gets fired.

https://learn.microsoft.com/en-us/windows-hardware/drivers/i...

The damage may have been done in a dependency which was not signed by Microsoft. Who knows? Hopefully we'll find out.

In general, a fair amount of the bad behavior of windows devices since Vista has been really about poorly written drivers misbehaving, so there appears to be value in that talking point. All the Vista crashes after release (according to some sources, 30% of all Vista crashes after release were due to NVidia drivers), notably, and more recently if you've ever tried to put your Windows laptop to sleep, and discovered when you take it out of your bag that it had promptly woken back up and cooked itself into having a dead battery. (Drivers not properly supporting sleep mode) WHQL has some things to answer for for sure.

It is their fault, not Microsoft's. The driver was fine.

Not questioning that it happened, but this was a boot loop after a content update. So if the computers were off and didn't get the update, and you booted them, they would be fine. And if they were on and you were using them, they wouldn't be rebooting, and it would be fine.

How did it happen that you were rebooting in the middle of treating a heart attack?

> And if they were on and you were using them, they wouldn't be rebooting, and it would be fine.

Windows has been notorious for forcing updates down your throat, and rebooting at the least appropriate moments (like during time-sensitive presentations, because that's when you stepped away from the keyboard for 5 minutes to set up the projector). And that's in private setting. Corporate setting, the IT department is likely setting up even more aggressive and less workaround-able reboot schedule.

Things like this is exactly why people hate auto-updates.

The assumption is that "pros" and "enterprise" either know how to use provided controls or have WSUS server setup which takes over all of scheduling updates.

Especially if an infected machine can attack others?

Users/IT regularly would never update or deploy patches which has its own consequences. There’s no perfect solution—but rather there to accept the pain.

It’s a lot like herd immunity in vaccines.

Yes. But you don't deploy experimental vaccines simultaneously across the entire population all at once. Inoculating an entire country takes months; the logistics incidentally provide protection against unforeseen immediate-term dangerous side effects. Without that delay, well, every now and then you'd kill half the population with a bad vaccine. The equivalent of what's happening now with CrowdStrike.

in the same way cars are notorious for forcing you to run out of gas while you're driving them and leaving you stranded... because you didn't make time to refill them before it became a problem.

> "Things like this is exactly why people hate auto-updates."

And people also hate making time for routine maintenance, and hate getting malware from exploits they didn't patch, and companies hate getting DDoS'd by compromised Windows PCs the owners didn't patch, and companies hate downtime from attackers taking them offline. There isn't an answer which will please everyone.

> "This prevention of functionality during a critical period while forcing an update would be like if a modern car refused to drive during an emergency"

Machines don't know if there's an emergency going on; if you don't do maintenance, knowing that the thing will fail if you don't, then you're rolling the dice on whether it fails right when you need it. It's akin to not renewing an SSL certificate - you knew it was coming, you didn't deal with it, now it's broken - despite all reasonable arguments that the connection is approximately as safe 1 minute after midnight as it was 1 minute before, if the smartphone app (or whatever) doesn't give you any expired cert override then complaining does nothing. Windows updates are released the same day every month, and have been mandatory for eight years: https://www.forbes.com/sites/amitchowdhry/2015/07/20/windows...

And we all know why - because Windows had a reputation of being horribly insecure, and when Microsoft patched things, nobody installed the patches. So now people have to install the patches. Complaining "I want to do it myself" leads to the very simple reply: you can - why didn't you do it yourself before it caused you a problem?

If you're still stubbornly refusing to install them, refusing to disable them, refusing to move to macOS or Linux, and then complaining that they forced you to update at an inconvenient time, you should expect people to point out how ridiculous (and off-topic) you're being.

> It's akin to not renewing an SSL certificate.

Your choice of analogies is a good one. I have done SSL type stuff since 1997.

Doesn't matter: I would have to work a few hours very carefully before modifying my web server config. And test it.

I am terrified by scale of deployment involved in this CloudStrike update.

In more professional settings than private small car ownership, you often will both have regular maintenance updates provided and mandates to follow them. Sometimes they are optional because your environment doesn't depend on them, sometimes they are mandatory fixes, sometimes they change from optional to mandatory overnight when previous assumptions no longer apply.

Several years ago a bit over 100 people and uncounted amount of possible more had their lives endangered because an extra airflow directing piece of metal was optional, and after the incident it was quickly made mandatory, with hundreds of aircraft being stopped to have the fix applied (which previously was only required for hot locations - climate change really bit it).

Similarly, when you drive your car and it fails to operate, that's just you. When it's a more critical service, you're either facing corporate, or in worst case, governmental questions.

If said update pushes you into bsod where automatic watchdog (by default set enabled in windows) reboots...well, here you have a bootloop

I can see very well how one computer could have screwed all others. It's really not hard to imagine.

[0] https://about.ascension.org/cybersecurity-event

Different medications may be pushed (injected into the patient) to help stabilize them. These medications are recorded via a bar code and added to the patients chart in Epic. Epic is the source of truth for the current state of the patient. So if that is suddenly unavailable that is a big problem.

Anyone involved in designing and/or deploying a system where an application outage threatens life safety, should be charged with criminal negligence.

A receipt printer in every patient room seems like a reasonable investment.

The grandparent indicated that the problem was that when all tow computers went down, they couldn’t look up what had already been done for the patient. I suggested a simple solution for that - receipt printers.

After the computers fail you tape the receipt to the wall and fall pack to pen and paper until the computers come back up.

I completely understand the scale of the outage today. I am saying that it was a stupid decision and possibly criminally negligent to make a life critical process dependent on the availability of a distributed IT application not specifically designed for life critical availability. I strongly stand by that POV.

Just so I understand what you are saying you are proposing that we drown our hospital rooms in paper receipt constantly. In the off chance the computers go down very rarely?

Do you see any possible drawbacks with your proposed solution?

> possibly criminally negligent to make a life critical process dependent on the availability of a distributed IT application

What process is not “life critical” in a hospital? Do you suggest that we don’t use IT at all?

Why does everything need the internet?

Why would you ever need to move a patient from one hospital room containing one set of airgapped computers into another, containing another set of airgapped computers?

And sending data out can be done quite securely. Then replies could be highly sanitized or kept on specific machines outside the air gap.

And now you've added an army of people running around moving USB sticks, or worse, printouts and feeding them into other computers.

It's madness, and nobody wants to do it.

In my eyes, there is a technical solution therr that keeps friction low for hospital staff: network stuff, on an internet, but not The Internet...

Edit: I've since been reading the other many many comment threads on this HN post which show the reasons why so much stuff in healthcare is connected to each other via good old internet, and I can see there's way more nuance and technicality I am not privy to which makes "just connect LANs together!" less useful. I wasn't appreciating just how much of medicine is telemedicine.

Yes there will be some pain, but the alternative is what we have right now.

> nobody wants to do it.

Tough luck. There's lots of things I don't want to do.

But they switched over to EMR because the advantages of Pyxis[1] in getting the right medications to the right patients at the right time- and documenting all of that- are so large that for patient safety reasons alone it wins out over paper. You can fall back to paper, it's just a giant pain in the ass to do it, and then you have to do the data entry to get it all back into EMR's. Like my wife, who was working last night when everyone else in her department got Crowdstrike'd, she created a document to track what she did so it could be transferred into EMR's once everything comes back up. And the document was over 70 pages long! Just for one employee for one shift.

1: Workflow: Doctor writes prescription in EMR. Pharmacist reviews charts in EMR, approves prescription. Nurse comes to Pyxis cabinet and scans patient barcode. Correct drawer opens in cabinet so the proper medication- and only the proper medication- is immediately available to nurse (technicians restock cabinet when necessary). Nurse takes medication to patient's room, scans patient barcode and medication barcode, administers drug. This system has dramatically lowered the rates of wrong-drug administration, because the computers are watching over things and catch humans getting confused on whether this medication is supposed to go to room 12 or room 21 in hour 11 of their shift. It is a great thing that has made hospitals safer. But it requires a huge amount of computers and networks to support.

Why would a Pyxis cabinet run Windows? I realize Windows isn't even necessarily at fault here, but why on earth would such a device run Windows? Is the 90s form of mass incompetence in the industry still a thing where lots of stuff is written for Windows for no reason?

This much of patients state will be carried on their wrist. Maybe for complex cases you need two stickers. Have to be judicious in encoding data, maybe just last 48 hours.

Handheld qr readers, off line that read and display QR data strings.

I'm not even beyond two degrees of seperation here. I don't think a court'll have trouble navigating it.

Cloudstrike did not even have a duty of care to their customer, let alone their customer’s customer (speaking for my jurisdiction, of course).

Now, a tort is less of a stretch than a crime, but thank goodness I’m not a lawyer so I don’t have to figure out what circumstances apply and how much liability the TOS and EULAs are able to wash away.

If I go to Home Depot to buy rope for belaying at my rock climbing center and someone falls, breaks the rope and dies, then I am on the hook for manslaughter.

Not the rope manufacturer, who clearly labeled the packaging with "do not use in situations where safety can be endangered". Not the retailer, who left it in the packaging with the warning, and made no claim that it was suitable for a climbing safety line. But me, who used a product in a situation where it was unsuitable.

If I instead go to Sterling Rope and the same thing happens, fault is much more complicated, but if someone there was sufficiently negligent they could be liable for manslaughter.

In practice, to convict of manslaughter, you would need to show an individual was negligant. However, our entire industry is bad at our job, so no individual involved failed to perform their duties to a "reasonable" standard.

Software engineering is going to follow the path that all other disciplines of meatspace engineering did. We are going to kill a lot of people; and every so often, enough people will die that we add some basic rules for safety critical software, until eventually, this type of failure occuring without gross negligence becomes nearly unthinkable.

Then again, the management that put this in are probably also the same idiots that insist on a 7 day lead time CAB process to update a typo on a brochure ware website "because risk".

Some hospitals require you to input this in order to even get physical access to the medications.

Although a crash cart would normally have common things necessary to save someone in an emergency, so I would think that if someone was truly dying they could get them what they needed. But of course there are going to be exceptions and a system being down will only make the process harder.

Yep, the business continuity boxes are basically minimally connected PDF archives of patient records "printed" multiple times a day.

Technologically it seems doable. Big enough order brings down the costs.

https://soldered.com/product/soldered-inkplate-5-5-2%e2%80%b...

Of course the real backup plan should be designed based on the actual needs, perhaps the whole system needs an "offline mode" switch. I assume they already run things locally, in case the big cable seeker machine arrives in the neighborhood.

There are plenty of drugs that can only be given in certain quantities over a certain period of time, and if you go beyond that, it makes the patient worse not better. Similarly there are plenty of bad drug interactions where whether you take a given course of action now is directly dependent on which drugs that patient has already been given. And of course you need to monitor the patient's progress over time to know if the treatments have been working and how to adjust them, so if you suddenly lose the record of all dosages given and all records of their vital signs, you've lost all the information you need to treat them well. Imagine being dropped off in the middle of nowhere, randomly, without a GPS.

Will outages like this motivate a backup paper process? The automated process should save enough information on paper so a switch over to paper process at any time is feasible. Similar to elections.

Maybe a handheld device for scanning in drugs or entering procedure information that stores the data locally which can then be synced with a larger device with more storage somewhere that is also 100% local and immutable which then can sync to online systems if that is needed.

there are backup paper processes, but they start fresh when the systems go down

If it was printing paper in case of downtime 24/7, it would be massive wasteage for the 99% of time system is up

If medical data were synced to the cloud but also stored on the endpoint devices and local servers, you’d have more redundancy. Obviously much more complexity to it but that’s what it would take. Epic as single source of truth means everyone is screwed when it is down. This is the trade off that’s been made.

That's a recipe for a different kind of disaster. I actually used Google Keep some years ago for medical data at home — counted pills nightly, so mom could either ask me or check on her phone if she forgot to take one. Most of the time it worked fine, but the failure modes were fascinating. When it suddenly showed data from half a year ago, I gave up and switched to paper.

More seriously, we need better purpose build medical computing equipment, that runs on it's own OS, and only has outbound network connectivity for updating other systems.

I also think of things like the old school "check list boards" that used to be literally built into the yolk of the airplane they were made for.

That doesn't help when the system goes down and you lose the record of all medications administered prior to having to switch over to the Sharpie.

Reality is complex.

And that's in the OR, where vitals are automatically captured. There just aren't enough computers to do real-time electronic documentation, and even if there were there wouldn't be enough space.

Its easier, faster, and more accurate than writing in my experience. We have a page solely dedicated to codes and the most common interventions. Got IO? I press a button and its documented with timestamp. Pushing EPI, button press with timestamp. Dropping an I-Gel or Intubating, button press... you get the idea.

The details of the interventions can be documented later along with the narrative, but the bulk of the work was captured real-time. We can also sync with our monitors and show depth of compressions, rate of compressions and rhythms associated with the continuous chest compression style CPR we do for my agency.

Going back to paper for codes would be ludicrous for my department. The data would be shit for a start. Hand writing is often shit and made worse under the stress of screaming bystanders. Depending on whether we achieved ROSC or not would increase the likelihood of losing paper in the shuffle

This wisdom is echoed in some religious practices that avoid complete reliance on modern technology.

Okay, how does that monitor work? Genuinely curious.

if it works, there's a lot more to be done to get the patient to stable.

Nurses hated it.

If they go and publish "According to hackernews user davycro ..." _then_ there's a problem.

Edit: Found it. https://www.youtube.com/watch?v=nS9nLvGMLH0&t=947s

“Loive from NPR news in Washington“

I can't believe they pushed updates to 100% of Windows machines and somehow didn't notice a reboot loop. Epic gross negligence. Are their employees really this incompetent? It's unbelievable.

I wonder where MSFT and Crowdstrike are most vulnerable to lawsuits?

1. Problem A happens, it’s pretty bad

2. A fix is rushed out very quickly for problem A. It is not given the usual amount of scrutiny, because Problem A needs to be fixed urgently.

3. The fix for Problem A ends up causing Problem B, which is a much bigger problem.

tl;dr don’t rush your hotfixes through and cut corners in the process, this often leads to more pain

Everything about it reeks of incompetence and gross negligence.

It’s the old story of the user and purchaser being different parties-the software needs to be only good enough to be sold to third parties who never neeed to use it.

It’s a half-baked rootkit part of performative cyberdefence theatrics.

That describes most of the space, IMO. In a similar vein, SOC2 compliance is bullshit. The auditors lack the technical acumen – or financial incentive – to actually validate your findings. Unless you’re blatantly missing something on their checklist, you’ll pass.

Any exception made to this checklist is reviewed by third parties that couldn't care less, bean counters, or those technically incapable of understanding the nuance, leaving only the large providers able to compete on the playing field they manufactured.

Ofc I realize windows is the only option for many orgs but that raises the question why doesn’t microsoft sell ”windows hardened” or such to cover the checklist?

This should be far more robust solution than third parties running amock in the kernel.

I am LMFAO at the entire situation. Somewhere, George Carlin is smiling.

This is the result of giving away US jobs overseas at 1/10th the salary

We were discharged at midnight by the doctor, the nurse didn't come into our exam room to tell us until 4am. I can't imagine the mess this has caused.

Or, we could build robust systems that can tolerate indefinite down time. Might cost more, might need more staff.

Pick one. I’ll always pick the one that saves human lives when systems go down.

2. Hospitals should not have executives.

3. Hospitals should be community funded with backstop by the federal government.

4. PE is a cancer - let the doctors treat it.

... and seconding all the best wishes for the mother involved. Do get well.-

Wishes for a speedy recovery to your mom!

I hope no one uses such single point of failure systems anymore. Especially CS. The same is applicable for Cloudflare as well! But at least, the systems will be functioning standalone and accessible in their case and could cause only netwide outage! (i.e., if the CF infra goes down!)

Anyways, who knows what is going to happen with such widespread vendor dependency?

The world gets reminded about the Supply Chain Attacks every year which is a good (but a scary) one that definitely needs some deep thinking...

Up for it?

That's an extra 4 hours of emergency room fees you ideally wouldn't have to pay for.

Why would Windows systems be anywhere near critical infra ?

Heart attacks and 911 are not things you build with Windows based systems.

We understood this 25 years ago.

Smarter people than us have already thought through this and the cost-benefit analysis said "connect it to a server"

Why not an internal only network for all the terminals to talk to a central server, then disable any other networking for the terminals? Why do those terminals need a browser where pretty much any malware is going to enter from? If hospitals are paying out the ass for their management software from epic/etc, they should be getting something with a secure design. If the central server is the only thing that can be compromised then when edr takes it down you at least still have all your other systems, presumably with cached data to work from

Same goes for electronic medical records. There are people who assign ICD-10 codes (insurance billing codes) to patient encounters. Often this is a second job for them and they work remote and typically at odd hours.

A modern hospital cannot operate without internet access. Even a medical practice with a single doctor needs it these days so they can file insurance claims, access medical records from referred patients and all the other myriad reasons we use the internet today.

This stuff isn't impossible to solve. Rather, the incentives just aren’t there. People would rather build an apparatus for blame-shifting than actually just building a better solution.

If modern medicine is dangerous and fragile because of network connected equipment then that should be fixed even if the way it currently works doesn’t allow it.

If you don't understand why it has to be networked with extremely bad fallback to paper, then I suggest working in healthcare for a bit before pontificating on how everything should just go back to the stone age.