I'm getting more up to date technical details from the regular media.

This outage looks to be huge.

Alas, using the internet has given us a lot of efficiency. The trade off is resilience. The entire global system is more brittle than ever but it what gave it such speed.

As per your point, the base infrastructure is working perfectly for better or worse!

Imagine it's 1998 and Norton push a new definition file that makes NAV think kernel32 is a virus. The only real difference today is that always-on means we all get the update together, instead of waiting for mum to get off the phone this evening.

We got an email this morning telling us none of our usual airlines could take bookings right now. That wouldn't have been much different in 1998, airline bookings have been centralised for my entire lifetime.

It's one vendor pushing out a bad update, and thousands of companies with no supply chain diversity.

The base infrastructure worked perfectly fine. The stuff on top of it, not so great.

Reality: the infrastructure took down itself.

Maybe a false positive? (100% speculation. That's the problem using closed source software)

> 7/18/24 10:20PT - Hello everyone - We have widespread reports of BSODs on windows hosts, occurring on multiple sensor versions. Investigating cause. TA will be published shortly. Pinned thread.

> SCOPE: EU-1, US-1, US-2 and US-GOV-1

> Edit 10:36PT - TA posted: https://supportportal.crowdstrike.com/s/article/Tech-Alert-W...

> Edit 11:27 PM PT:

> Workaround Steps:

> Boot Windows into Safe Mode or the Windows Recovery Environment

> Navigate to the C:\Windows\System32\drivers\CrowdStrike directory

> Locate the file matching “C-00000291*.sys”, and delete it.

> Boot the host normally.

You do have your bit locker recovery key, right? .....right?

In Enterprise setups the key should be backed somewhere in Active Directory.

> A colleague is dealing with a particularly nasty case. The server storing the BitLocker recovery keys (for thousands of users) is itself BitLocker protected and running CrowdStrike (he says mandates state that all servers must have "encryption at rest").

> His team believes that the recovery key for that server is stored somewhere else, and they may be able to get it back up and running, but they can't access any of the documentation to do so, because everything is down.

One of my biggest frustrations with learning networking was not being able to access the internet. Nowadays you probably have a phone with a browser, but back in the day if you were sitting in a data room and you'd configured stuff wrong, you had a problem.

Or worse, if it's like where I worked in the past, they're still in the buying process for a safe (started 13 months ago) and the analysts are building up a general plan for the management of the safe combination. They still have to start the discussions with the union to see how they'll adapt the salary for the people that will have to remember the code for the safe and who's gonna be legally responsible for anything that happens to the safe. Last follow-up meeting summary is "everything's going well but we'll have to modify the schedule and postpone the delivery date of a few months, let's say 6 to be safe"

When we later moved to new offices, somebody found a solution that involved a 'stair-walking' device that could supposedly get the safe down to the ground floor. This of course jammed when it was halfway down the stairs. Hilarity ensued.

1. Boot the affected machine from the Windows installer disk

2. Use "Repair options"

3. Click through to the option to spawn a shell

4. It will now ask you for unlocking the disk with a recovery key. SKIP THAT.

5. In the shell, type: "manage-bde -unlock C: -Password", enter the password

6. The drive is unlocked, now go and execute whatever recovery you have to do.

Good luck.

Can you even get the secret from the TPM in recovery mode?

Given that you can (relatively trivially) sniff the TPM communication to obtain the key [1], yes it should be possible. Can't verify it though as I've long ago switched to Mac for my primary driver and the old cheesegrater Mac I use as a gaming rig doesn't have a hardware TPM chip.

[1] https://pulsesecurity.co.nz/articles/TPM-sniffing

I don't know how common it is to disable TPM-stored keys in companies, but on personal licenses, you need group policy to even allow that.

Although this is moot if Windows recovery mode is accepted as the right system by the TPM. But aren't permissions/privileges a bit neutered in that mode?

https://x.com/attilabubby/status/1814216589559861673

https://x.com/nathanmcnulty/status/1785094215749476722?s=46

GPO to fix:

https://gist.github.com/whichbuffer/7830c73711589dcf9e7a5217...

> Summary

> CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.

> Details

> Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.

> Current Action

> Our Engineering teams are actively working to resolve this issue and there is no need to open a support ticket.

> Status updates will be posted below as we have more information to share, including when the issue is resolved.

> Latest Updates

> 2024-07-19 05:30 AM UTC | Tech Alert Published.

> Support

> Find answers and contact Support with our Support Portal

It includes PDFs of some relevant support pages that someone printed with their browser 5 hours ago. That's probably the right thing to do in such a situation to get this kind of info publicly available ASAP, but still, oof. Looks like lots of people in the Reddit thread had trouble accessing the support info behind the login screen.

So engineer-like.

A Crowdstrike update being able to blue-screen Windows Desktops and Servers.

Whilst Crowdstrike are going to cop a potentially existential-threatening amount of blame, an application shouldn't be able to do this kind of damage to an operating system. This makes me think that, maybe, Crowdstrike were unlucky enough to have accidentally discovered a bug that affects multiple versions of Windows (ie. it's a Windows bug, maybe more-so than it is a Crowdstrike bug).

There also seems to have been a ball-dropped in regards to auto-updating all the things. Yes, gotta keep your infrastructure up to date to prevent security incidents, but is this done in test environments before it's put into production?

Un-audited dependence on an increasingly long chain of third-parties.

All the answers are difficult, time consuming, and therefore expensive, and are only useful in times like now. And if everyone else is down, then there's safety in the crowd. Just point at "them too", and stay the path. This isn't a profitable differentiation. But it should be! (raised fists towards the sky).

It doesn't operate in user space, they install a kernel driver.

And therein lies the problem!

You can, for instance, ask a running kernel if it is "tainted" by having loaded a non-GPL module.

I can, considering that you can do that from user space using strace. Or ebpf which is probably the actual right way to do this kind of thing.

And I bring up the distinction because while compliance is "sometimes" about safety, it's also very often about KPIs of particular individuals or due to imaginary liability for having not researched every possible "compliance" checkbox conceivable and making sure it's been checked.

Some computer security software is completely out of hand because its primary purpose is to have the appearance of effectiveness for the exec whose job is to tick off as many safety checkboxes as they can find, as opposed to being actually pragmatically effective.

If the same methodologies were applied to car safety, cars would be so weighed down by safety features, that they wouldn't be able to go faster than 40km/h.

Should you upgrade before 24.04.1 is released? It's scheduled for August 15.

Most things that third-party kernel drivers used to do (device drivers, file systems, etc) are now done just as well, and much more safely, in userspace. I'm surprised if Microsoft isn't heading in this direction too?

Presumably, Crowdstrike runs on macOS without a kernel extension?

That's correct: CrowdStrike now only installs an "Endpoint Security" system extension and a "Network" system extension on macOS, but no kernel extension anymore.

This is a syscall used by userspace to tell the kernel which memory portion is allowed to do syscalls

This syscall can only be used once : once the linker has done it, the kernel will refuse extra calls (so allowing more memory pages is not possible)

Second, still, that doesn't change anything. You can make your malware jmp to anywhere so that the syscall actually comes from an authorized page.

In fact, in windows environment, this is actively done ("indirect syscalls"), because indeed, having a random executable directly calling syscalls is a clear indicator that something is malicious. So they take a detour and have a legitimate piece of code (in ntdll) do the syscall for them.

As PC got faster, Microsoft could have returned to the microkernel architecture, or at least focused on isolating drivers better.

https://learn.microsoft.com/en-us/windows-hardware/drivers/i...

Point of information. "Guarantee" and "any" are unsubstantiated by that MS article.

> an ELAM driver can't be guaranteed to load before another ELAM driver of course,

Thanks for the correction.

Antivirus software by its nature probably needs the kind of access that would let it bluescreen your computer.

And there's a certain amount of sense to that, it has to get "under" the layer that viruses can typically get to, but I still think there should be another layer at which the OS is protected from misbehaving anti-virus software (which has been known to happen).

I understand I'm yelling into the storm here, because anti-virus also requires that level of system access due to the nature of what it's trying to detect. But then again, does it only need Ring 0 access for the worst of the worst? Can it run 99% of the time in Ring 1, or user space, and only instantiate it's Ring 0 privileges for regular but infrequent scans or if it detects something else may be 'off'?

Default Ring 0? Earn it.

This turns into a "what's your threat level" discussion.

https://en.wikipedia.org/wiki/Protection_ring#Miscellaneous

Originally, x86 processors had 4 levels of hardware protection, from ring 0 up to ring 3 (if I remember right). The idea was indeed that non-OS drivers could operate at the intermediate levels. But no one used them and they're effectively abandoned now. (There's "level -1" now for hypervisors and maybe other stuff but that's besides the point.)

Whether those x86 were really suitable or not is not exactly important. The point is, it's possible to imagine a world where device drivers could have less than 100% permissions.

While virus scanners might pick up some threats not addressed by OS updates yet every one of them I've seen is a rootkit in disguise wanting full system privileges. There are numerous incidents with security holes and crashes caused by these security products. They also aren't that clever: repeatedly scanning the same files 'on access' over and over again wasting CPU and IO is not going to give you any extra security.

CS has official RCE root/admin access on all the clients. Which skips any normal auth of the OS. Yes, on all windows, mac and linux.

Where you're correct is that it's on the administrators to rollout the updates, but I'm not sure that's how Crowdstrike works. It's a managed solution and updates are done for you, maybe that can be disabled, but I honestly don't know.

Yeah, this is what surprises me. Corporate infrastructure policy seems to have been matched to smart phone default settings.

These people report to the Board Chairman, don't understand any real implication of their work, and believe the world is a simplistic Red - Amber - Green grid.

I understand most CIOs / CTOs / CISOs in Corporate would buckle.

CS is not sold to SA or technical types. It's sold to management as a risk reduction.

The whole point is that if you are technical, you are so untrusted that management is willing to require circumvention of known good practices and force installation of this software against technical advice.

Concerns about usability between Windows and Linux in the modern day are disingenuous at best and malicious at worst. There is no UX concern when everything runs off a webapp these days.

Just use Linux. You will save money and time, and your system will be supported for many years, you won't be charged per E-Core, you won't suffer BSoDs in 2024. Red Hat is a trustworthy American company based out of Raleigh, NC, in case you have concerns of provenance.

Really there's no downside. If you were building your own company you would base your tech stack on Linux and not Windows.

Critical systems cannot go down; therefore they cannot run Windows. If they do, they are being mismanaged and run negligently. Management should have no issue finding Linux engineers, they are everywhere. I could make a killing right now as a consultant going from company to company and just swapping out Windows backends for Linux. And quite frankly I might just do that, literally starting right now.

Every system gets attacked, but I think your point shows that even with state-level attacks Linux handles it better than other platforms.

I'd also say this wasn't a good example of 'linux handling it better': usually when a mess like this occurs on windows all the corps get a quiet tap on the shoulder that they need to immediately patch when MS releases it, then a few days later it hits the news. In XZ's case, the backdoor was published before the team knew about it, huge mess.

> all the corps get a quiet tap on the shoulder that they need to immediately patch when MS releases it, then a few days later it hits the news

AFAIK, distros were notified and released a patched version of xz like a week before it hit the news, so at least a lot of machines received it via automatic updates.

I would say issue 1 is management/compliance forcing admins to install malwares like crowdstrike. But issue 1 is because of issue 2 which is about admins / app devs / users aren't smart enough to not have their machines compromised on a regular basis in the first place. And issue 2 is because issue 3 of the software industry not focusing on quality and making bug free software.

All in all this should be mitigated by more diversity in OS, software and "said security solution". Standardization and monopolies works well until they don't and you get this kind of shit.

But would it crash the OS?

Falcon sensor is the most CPU intensive app running in the cluster and produces a constant stream of disk activity (more so than any of our apps).

It hasn't crashed anything yet but it definitely leaves me feeling iffy about running it.

I don't like CrowdStrike at all. I got contacted by our security department because I used curl to download a file from GitHub on my dev box and it prompted a severe enough security warning that it required me to explain my intent. That was the day I learned I guess every command or maybe even keystroke I type is being logged and analyzed.

An issue might not be as universal as on windows, because some distros do things differently like not using glibc, or systemd, or whatever. Yet there are some baselines common to the most popular ones.

Windows actually runs a lot of drivers in user-mode, even GPU drivers. largely this is because third-party drivers were responsible for the vast majority of blue screens, but the users would blame Microsoft. which makes sense; Windows crashes so they blame Windows, but I doubt anyone blamed Linux for the kernel panic.

Maybe this is just my ignorance about windows and its ecosystem but it seems most admins this morning were clueless on how to fix that automatically and remotely on n machines and would resort to boot in safe mode and remove a file manually on each single server. This is just insane to think that supposed windows sysadmins / cloudops have no idea how to deploy a fix automatically on that platform.

https://access.redhat.com/solutions/7068083

https://forums.rockylinux.org/t/crowdstrike-freezing-rockyli...

Companies like CS live on reputation, it should be dragged down.

Granted it didn't down linux this time but nothing is stopping it.

My work laptop is running Ubuntu, and corporate IT requires Symantec Antivirus to be running on it

It is an incredibly frustrated battle akin to Sisyphus.

Only if you don't need a GUI/Desktop.

I not only need a GUI/Desktop, it's my daily driver!

And there are precious few things that Windows GUI/Desktop provides which I don't have on Linux, while the reverse is never true.

When I used Mac (Big Sur, I think?) until a year ago, I was absolutely miserable about having to use such a primitive GUI.

I switched critical systems to illumos and BSD years ago and it's been smooth sailing ever since. Nowadays there really is no need to contribute to linux monoculturization whatsoever.

> Linux being the most advanced operating system in the world without question.

Very strong and mostly unfounded claim; there are specific aspects where Linux is "more advanced", and others where Windows come out ahead (e.g. almost anything related to hardware-based security and virtualization).

> your system will be supported for many years

Windows Server 2008 was supported until earlier this year, longer than any RHEL release.

> you won't suffer BSoDs in 2024

Until you install a shitty driver for a dubious (anti)malware service.

https://www.computerworld.com/article/1642872/q-a-microsoft-...

https://betanews.com/2006/10/18/mcafee-ms-failing-to-provide...

This is due to their price making them relatively more available to the enthusiasts than say Hellcats, enthusiasts who may not be experienced enough to deal with having that much power available to them in a RWD car. This confluence of power, confidence and lack of skill often comes to a head when the enthusiast goes to a car meet to show off and meet with like minded folks. At the conclusion of the meet, or during a group drive, they'll often pull a sick burnout as they pull out of the parking lot on to a street.

A sick burnout they haven't practiced, and will often cause them to lose the back end sending the car into the curb, a tree, or a crowd of like minded attendees at the car meet. Therefore, the reputation.

For example: https://www.youtube.com/watch?v=DPx5aBI8UTQ

Also heard today crowd stroke.

He actually threatened to fire us as a client because he claimed he didn’t want the CS brand associated with an org that wasn’t “fully protected” by CS. By far the worst vendor contact I’ve ever had. I’ve had nicer meetings with Oracle lawyers than I was having with this guy. I hope this sort of thing humbles them a little.

Evidence is pointing towards him actually being right about this, despite likely being wrong about everything else.

It'd be worth giving him a call, just to check in how he's going, and take him up on the offer to fire you as a client.

Hopefully not. It would be better that this company is sued into oblivion by all the customers that were affected by this huge outage.

If their mission is to protect businesses they should understand your concerns.

Speed is useless without control.

We had a buggy client release during the rollout which consumed all the CPU in one of our test environments (something he assured us could never happen), and he calmed down a bit after that. Prior to that though he was doing stuff like finding our CISO on LinkedIn to let him know how worried he was about our rollout pace, and that without CS protection a major breach could be imminent.

For example, the way Oracle's lawyers are known.

What I hope, is that they stop to exist as a product and as a company. They have caused inconvenience, economic damage in global scale and probably also loss of life, given that many hospitals, ER units had outages. It has been proven that their whole way of working is wrong, from the very foundation to the top.

sounds like they didnt test all cases and stumbled on a windows bug

https://www.nzherald.co.nz/nz/bank-problems-reports-bnz-asb-...

Banks are down so petrol stations and supermarkets are basically closed. People can't check in to airline flights, various government services including emergency telephone and police are down. Shows how vulnerable these systems are if there's just one failure point taking all those down.

We did not pick them, and cancelled any other relashionship we had with them, in IR space by example.

https://www.crn.com/news/security/crowdstrike-poaches-two-ex...

Currently seeing this happening in real time in the UK.

Surprisingly: banks, government, police, fire department, railways, buses are mostly unaffected.

Maybe they have a good IT department/provider.

At least the central flight booking system is up I guess. Google brought it years ago and it's a mainframe.

Hence why google flights is so tapped in :)

Update: 911 is down in Oregon too, no more ambulances at least.

More seriously I am not saying you should run some critical services on menuetos or riscos but the BSDs are still alive and kicking as well as illumos and its derivatives. And yes I think a bit of diversity allows some additional resilience. It may necessitate more workforce but imho it is worth the downsides.

So if you have truly mission critical systems you should probably have more have at least 2 significantly different systems, each of them being able to maintain some emergency operations independently. Doing this with 2 Linux distros is easier than doing it with Linux and Windows. For workstations Macs could considered, for servers BSD.

Probably many companies will accept the risk that everything goes down. (Well, they probably don't say that. They say maintaining a healthy mix is too expensive.)

In that case you need a clearly phased approach to all updates. First update some canaries used by IT. If that goes well update 10% of the production. If that goes well (well, you have to wait until affected employees have actually worked a reasonable time) you can roll out increasingly more.

No testing in a lab (whether at the vendor or you own IT) will ever find all problems. If something slips through and affects 10% of your company it's significantly different from affecting (nearly) everyone.

The ideal would be to do canary rollouts (1%, then 5%, 10% etc.) to minimise blast radius, but I guess that's incompatible with antiviruses protecting you from 0-day exploits.

Not saying CS doesn't care enough but what may be a minor update to the team that did this and not necessary for a slow rollout is actually something that really should be supervised in that way.

Except, unknown to us, our virtualization provider had a massive infrastructural issue at exactly that moment preventing VMs from booting back up... That wasn't a fun night to failover services into the secondary DC.

The issue is update rollout process, lack of diversity of these kind of tools in the industry, and the absolute failure of the software industry to make decent software without bug and security holes.

It has happened before where security software has unwanted effects, can't say i remember anyone else managing to blue screen Windows and require a safe mode boot to fix the endpoints though.

"711 has been affected by the outage … went in to buy a sandwich and a coffee and they couldn’t even open the till. People who had filled up their cars were getting stuck in the shop because they couldn’t pay."

Can't even take CASH payment without the computer, what a world!

Depends if the retailer wants to take the chance of all that.

There are stores in many places in the country with sporadic internet or where outages are not uncommon, and where you would want to configure the terminals to still work while offline. In these cases, the payment terminals can be configured to take offline transactions, and they are stored locally on the lane or a server located in the store until a connection to the internet is re-established.

Luckily I can just give someone a paper wallet containing crypto. No transactions, no traceability, no rules.

They use Windows as a part of their server infrastructure?

I went into our crowdstrike policies and disabled auto update of the sensor. Hopefully this means it doesnt hit everything. Double check your policies!!!

Edit:

Crowdstrike has an article out on the manual fix:

https://supportportal.crowdstrike.com/s/article/Tech-Alert-W...

Of course, no news on its websites or socials because those people didn’t start until 9am.

I think they finally fixed that.

So a normal day at SeaTac?

1.enter in drive C: 2.system 32 folder 3. Drivers 4. Rename crowdstrike folder to something else doesent matter what.

https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/bsod_e...

Looks like this is a big deal.

https://www.geekersdigest.com/windows-blue-screen-crash-caus... https://www.timesnownews.com/technology-science/latest-crowd...

This is gonna be a tough Friday for IT departments...

Also in Australia

Philosophically it's always good to have diversity, precisely to avoid such disruptions. But the real issue here is: A) Apparently half the world runs CloudStrike... So everything is disrupted. B) Apparently CloudStrike didn't test their update properly.

I'm very curious what will happen to CloudStrike. This seems like a huge liability?

Delta is going after CrowdStrike, and I doubt they'll be the last.

https://x.com/s0lst1c3/status/1814182835269337548

But yeah I've seen US companies for example only doing their initial releases in the USA only which has zero value for issues that might appear with different localization/language settings for example.

DELL laptops are observed, after blue dump . Server is up and running fine

Temporary workaround leads compliance issue.

From India

Did it work for you?

https://abcnews.go.com/US/american-airlines-issues-global-gr...

The FAA is telling air traffic controllers to tell airborne pilots that airlines are currently experiencing communication issues.”

Open the Command Prompt as an administrator. Run the following command to uninstall the current version: shell

sc delete csagent

1. Rename csagent.sys ( the file causing the BSOD ) 2. Rename c:\windows\System32\Drivers\Crowdstrike

Again, do this at your own risk. Both workaround have been reported as "working". I am Linux user so I cannot tell.

Edit: took out a bit of snark.

If I am being naive here, happy to hear other opinions. But I hate opening my company Windows laptop and having the fans turn to 11 just because some "security" software is parsing random files for malicious signatures or running an update that BSOD loops.

Thank fuck Netflix runs on Linux. I just hope the full chain from my TV to Netflix is immune...

(Some of the team members lurk around here, maybe we'll get lucky and one comments)

Which makes it less likely to take a 3rd party agent from a snakeoil company that sells to execs, then embed it at low levels into mission critical services with elevated privileges, then give it realtime external updates that can break your platform at any point.

I work in hardware development and such a failure is almost impossible to imagine. It has to work, always. It puzzles me why this isn't the casebfor software. My SWE colleagues often get mad at us HW guys because we want to see their test coverage for the firmware/drivers etc.. The focus is having something which compiles and pushing the code to production as fast as possible and then regressing in production. Most of HW problems are a result of this. I found it's often better to go over the firmware myself and read line by line to understand what the code does. It saves so much time from endless debugging sessions later. It pisses of firmware guys, but hey, you have to break some eggs to make an omelette.

In my anecdotal experience, its because corporate software projects are not typically run by people who are good at building safe things - but rather, just building things quickly.

There's a huge issue with the mentality of "it works, ship it" being propagated.

I build systems software for safety-critical and mission-critical markets, and I can say without a doubt that if there aren't at least two quality stages in your process workflow (and your workflow isn't waterfall), then you're going to be in for a rough time, rookies.

Always, always delay your releases, and always, always, eat your own dog food by testing your delayed releases in your own customer-like environment, which is to say, never release a developers build ..

When the entire society and economy are being digitized AND that digitisation is controlled and passes through a handful of choke points its an invitation to major disaster.

It is risk management 101, never put all your digital eggs in one (or even a few) baskets.

The love affair with oligopoly, cornered markets and power concentration (which creates abnormal returns for a select few) is priming the rest of us for major disasters.

As a rule of thumb there should be at least ten alternatives in any diversified set of critical infrastructure service providers, all of them instantly replaceable / forced to provide interoperability...

Some truths will hit you in the face again and again until you acknowledge the nature of reality.

Can you imagine having just one road connecting two big cities to cut costs? No alternative roads, nor big nor small.

That will be really cheap to maintain, and you can charge as much as you want in tolls as there are no alternatives. And you can add ads all over the road as people has to watch them to move from one city to the other.

And if the road breaks, the goverment needs to pay for the cost as they cannot allow the cities to go unconnected.

We live in the middle-ages of technology.

Maybe this crowdstrike outage will be the "burning building" that will bring change :)

Only when you are focusing on short-term effects. If you thinking long-term it is always better to not be out of business.

Sure! But now imagine those roads private property.

The whole startup/unicorns ecosystem exists only because there is the possibility of becoming the dominant player in a field within a few years (or being bought out by one of the big players). This "love affair with oligopoly" is the reason why Ycombinator/HN exists.

Software debt is networked.

I'm writing this in the wake of the aftermath of the disclosure of the log4j zero-day vulnerability. But this is only a recent example of just one kind of networked risk.

With managed services we effectively add one more level to the Inception world of our software organisation. We outsource nice big chunks of supply chain risk management, but we in-source a different risk of depending critically on entities that we do not control and cannot fix if they fail.

Not to mention the fact that change ripples through the parallel yet deeply enmeshed dimensions of cyberspace and meatspace. Code running on hardware is inexorably tied to concepts running in wetware. Of course, at this level of abstraction, the notion applies to any field of human endeavour. Yet, it is so much more true of software. Because software is essentially the thoughts of people being played on repeat.

It's a swinging pendulum of decisions. And developers know that software/hardware provision is a house of cards. The more levels of dependency, the more fragile the system is.

Absolutely horrible when lives will be lost, but determining the way our global systems are engineered and paid for will always be a moving target based on policy and incentive.

My heart goes out to life and death results of this. There are no perfect tech solutions.

Googling it seems crowdstrike has a history of causing kernel panics.

https://www.google.com/search?q=crowdstrike+kernel+panic

This is checkbox software, selling to decision makers in enterprises who will never see nor touch the software they are buying.

This kind of software where the user isn’t the customer is always terrible.

Never heard this name before. Thx.

Everytime there was a mysterious performance problem affecting a random subset of machines, it was Tanium. I know how difficult it is for anyone to just get rid of this type of software, but frankly it has been proven over and over that antivirus are just more surface attack, not less.

This CS thing, there’s no way around. You use it and they screw up, you get hit. Period. You don’t failover to another data center in Europe or Asia. You just go down.

Hardware, even cloud hardware, is rarely the issue. Probably especially cloud hardware is not an issue because failover is so inexpensive relative to software.

Software is a different issue entirely. How many of us will develop, shadow run, and maintain a parallel service written on a separate OS? My guess is “not many”. That’s the redundancy we’re talking about to avoid something like this. You’d have to be using a different OS and not using CS anywhere in that new software stack. (Though not using CS wouldn’t be much of a problem if the OS is different but I think you see what I mean.)

Amazon, implementing failover for your hardware is a few clicks. But if you want to implement an identical service with different software, you better have a spare dev team somewhere.

AWS is, as any company, centralized in a way or another.

Want to be sure you won't be impacted by AWS East going down, even if you run in another region? Well, better be prepared to run (or have a DRP) on another cloud provider then...

The cost of running your workload on two different CSP is quite high, especially if your teams have been convinced to use AWS-specific technologies. You need to first get your software stack provider agnostic and then manage the two platform in sync from a technical and contract perspective, which is not always easy...

Why multiply your problems? Use your cloud service provider only to access hardware and leave the rest of that alone. That way any cloud provider will due. Any region on any cloud provider will due. You could even just fallback to your own racks if you want. Point is, you only want the hardware.

Now to get that level of redundancy, you would still have to create 2 different implementations of your application on 2 different software and OS stacks. But the hardware layer is now able to run anywhere. Again, you can even have a self hosted rack in your dispatch stack.

So hardware redundancy is easy to do at the level the original commenter recommends . Software redundancy is incredibly difficult and expensive to do at the level the original commenter was talking about. Your idea to make a hardware/cloud abstraction layer only multiplies the number of software layers you would need multiple implementations of, shadow run and maintain to achieve the hypothetical level of redundancy.

The fact it's widespread is because so many individual organisations individually chose to use CrowdStrike, not because they all got together and decided to crown CrowdStrike as king, surely?

I agree with you in principle, but the only solution I can think of would be to split up a company with reach like CrowdStrike's. The consequences of having to do that are up for debate.

You're down? Great, so are your competitors, your customers, and your suppliers. Head to the pub. Actually, you'll probably get more real value there, as your competitors, customers and suppliers are at that same pub. Insurance multinationals have been founded from less.

That didn't affect any OT though, so it was more just proof that 90% of work carried out via O365 adds no real value. Knowing where the planes are probably is important.

I mean yeah, that's the other thing - the Keynesian sound banker aspect. But that's more for software that you're intentionally using for your business processes. I don't think anyone was thinking about Cloudstrike being down in the first place, unless they were worried about an outage in the webpage that lists all the security certifications they have.

Yes, there could be reasons to choose a lesser-known product, but they better be really good reasons.

Because there are multiple general reasons in the other direction, and incidents like this are actually one of those reasons: they could happen with any product, but now you have a bigger community sharing heads-ups and workarounds, and vendor's incident response might also be better when the whole world is on fire, not only a couple of companies.

It's that so many people are on Azure - which is a defacto monopoly for people using Microsoft stack - which is a defacto monopoly for people using .Net

And if they're doing that, the clients are on Windows as well, and probably also running Crowdstrike. The AD servers that you need to get around Bitlocker to automatically restore a machine are on Azure, running Windows, running Crowdstrike. The VM image storage? Same. This is basically a "rebuild the world from scratch" exercise to some greater or lesser degree. I hope some of the admins have non-windows machines.

> I agree with you in principle, but the only solution I can think of would be to split up a company with reach like CrowdStrike's.

Changing corporate structures doesn't necessarily help. It's possible that if CrowdStrike were split up into to smaller companies, all the customers would go to the one with the "better" product and we'd be in a similar position.

So here we are.

Most of crap security I met in big organisations was driven by checklist audits and compliance audits by a few "security" firms. Either you did it the dumb way or good luck fighting your org and their org to pass the audit.

The problem is allowing a single vendor, with a reputation of fucking up over and over again, to push code into your production systems at will with no testing on your part.

But no, they are okay with a black box that calls home and they give it kernel access to their machines. What?

(I am operations management and fought against this product and approach for months)

If his business is the most critical thing for people's lives, sure. If it's anything else, his customers will have other things to worry about.

Not everything needs to be done through a computer, and we are seeing the effects now of organizing our systems such that the only way to interface with them is through a digital device or a smartphone, with no alternative. Such are the consequences of moving everything "into the cloud" and onto digital devices as a result of easy monetary policy and the concomitant digital gold rush where everyone and their dog scrambled to turn everything into a smartphone app.

Being prepared can cost more and/or be less flashy (read: I didn't get touch-only) but it's only peace of mind, at least for critical components. I want a thermostat that works, I don't get no satisfaction from any bragging rights. Nod to the Rolling Stones.

Have home automation? Sorry, you'll have to use the Internet.

I vote with my dollars, so it cost them the higher-margin sale. I also went with the mid-tier system, and grabbed a Z-Wave compatible thermostat along with it. I wonder if I'll miss the nifty variable-speed system?

I really wish everyone would stop trying to trap us into their walled gardens. Apple at least lets people write software for theirs, but the hardware/appliance manufacturers (not to mention the automotive folks) are awful about this.

*And* adding on a subscription.

It definitively matters. The main issue he is that Crowdstrike was able to push and update on all server around the world where all their agent is installed ... it looks like an enormous botnet ...

We need a detailed post-mortem on what happened here.

The cost and benefits of managing risks need to be balanced.

So I am not convinced that there need to be "at least ten alternatives" to be fail safe as society.

Where I agree with you is that these decisions should be deliberate and done after a cost / benefit analysis.

The required number "N for safety" is a good discussion to have. Risk-Return, Cost-Benefit etc are essential considerations. We live in the real world with finite resources and stark choices. But I would argue (without trying to be facetious) that they are risk management 102 type considerations.

Why? Because they must rely on the pretense of knowledge [1]. As digitization keeps expanding to encompass basically everything we do, the system becomes exceedingly complex, nobody has a good picture of all internal or external vulnerabilities and how much they might cascade inside an interconnected system.

Assessing cost versus benefit implies one can reasonably quantify all sides of the equation. In the absence of a demonstrably valid model of the "system" the prudent thing is to favor detail-agnostic rules of thumb. If these rules suggest that reducing unsafe levels of concentration is not economically viable there must be something wrong with the conceptual business model of digitization as it is now pursued.

[1] https://www.nobelprize.org/prizes/economic-sciences/1974/hay...

In this case, has CS done any QA on this release? Have they tested it for months on all the variations of the devices that they claim to support? It seems not.

We are going to get hit by some terrible outage eventually (I hope someone is tracking things like what happens if a big war breaks out and the GPS constellations all go down together). But having 10x providers won't help against the big IT-related threats which are things like grid outages and suchlike having cascading effects into food supplies.

I mean, plenty of businesses only have penguin eggs in their basket, and some sort of penguin problem would cause major problems for them. I believe that last time this happened was with the leap second thing around 2005 or thereabouts.

"Don't put all your eggs in one basket" sounds nice, but it would mean a completely different independent service all through your stack. That's not really realistic, IMHO.

The bigger issue here is that: 1) some driver update "just" gets pushed (or how does this work?), and 2) the inability to easily do "this is broken, restore to last version". That is even something that could be automatic.

The fact that physical distance is no longer a limit to who you do business with means that you can select the cheapest vendor globally, but then that vendor has an incentive to hyperspecialize (because everyone goes to them for this one thing), which means that even more people go to them.

Avoiding once-in-a-century events just isn't something we're willing to pay the extra cost for, except now we have around twenty places where these once-in-a-century events can happen, which kind of makes them more frequent.

How much stuff do you host on Hetzner instead of AWS?

I mean how did this happen at all? Are there no checks in place @ crowdstrike? Like deploying the new update to a selected machines and check whether everything is ok, and then releasing it to the wild incrementally?

Mind boggling.

But they do matter. This is elementary. It's like saying "playing with matches doesn't matter". This is a problem that has happened before, albeit on smaller scale, and the solution/cure is well known and imho it should be established 2 decades ago to every org on the planet.

This is basic COBIT (or BYOFramework) stuff from 10-15-20 years ago.

How can you push a patch/update without testing it fist? I get it if you are a tiny company with 1 IT person an 20 local PCs. Stuff like that cripples you for a couple of days. But when you are an org, with 10k+ laptops, 500+ servers (half of them MS Win), how can you NOT test each and every update?

If you don't want to have the test/staging environments, then at least wait 1-3-5 days to see what the updates will do to others/the news.

Sorry not sorry guys and gals. I've been auditing systems and procedures for so many years, that this is a basic failure. "One cannot just push an update without testing it first" any update, no matter how small/innocent.

Multiple machines, each one using different vendor software? What other software needs to be partitioned this way? What about combinations of this software?

I’m just barely awake but don’t know if I’m affected yet. One of my devs is, our client support staff is, and I have no idea how our servers are doing just yet.

And does anyone actually know how to actually implement this, at the scale required (dealing with billions of transactions daily) in a way that would resolve the problems we are seeing?

It very much seems like a data access problem; places can't access/modify data. The physical disks themselves are most likely fine, but the 'interfaces' are having troubles (assuming that the data isn't stored on the devices having the issue).

But in any case how do you design a system that, if the 'main' interface is troubled, you can switch over, instantly, seemlessly, duplicating access controls, permissions, data validation, logic etc.

There is a reason everything is centralised because it makes no financial sense to duplicate for an extremely unlikely and rare chance. The world is random and these things will happen, but a global outage on this type of scale is not a daily occurance.

We'll look back in a few years and think "those were a crazy few hours" and move on...

I think you mean 14. https://xkcd.com/927/

> When the entire society and economy are being digitized AND that digitisation is controlled and passes through a handful of choke points its an invitation to major disaster.

Once again, it's Microsoft, directly, or indirectly, choosing a strategy of eventually getting all worldwide Windows desktops online, and connected via their systems.

Which is why I installed Fedora after Windows 7 and never looked back. 100% local, 100% offline if needed.

My company is looking to a non-Microsoft desktop. We're not affected by this, but it will certainly encourage us to move sooner rather than later.

These small 'downages' basically never cause serious issue. Your solutions are just alarmist and extremely costly (though they will provide developer employment...).

Hospitals, airlines, 911, grocery stores, electric companies, gas companies, all offline. There will be more than a few people dead as an indirect result of this outage, depending on how long it lasts.

Emergency Departments and 911 were knocked offline. People will indirectly die because of this, just like the last time 911 went down, and just like the last time EDs went down.

It's not alarmist, it's realist.

> Society was able to move to mass WFH on a global scale in a single month during Covid

I don't know how much WFH saved lives, seeing as ordered isolation and social distancing was a thing during the Spanish Flu too (you just take the economic hit). But yes it allowed companies to keep maintaining profits. Those that couldn't WFH got paid in most countries anyway (furlough in England, etc).

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2862334/

If only because we can then move away from one-size-fits-all, while mitigating the short-term impact of events like the above.

This problem is pervasive and effects everything from food supply (planting genetically identical seeds rather than diversified "heirloom" crops) to businesses across the board buying and gutting their competitors thus reducing consumer choice.

It's a tough problem akin to a multi-armed bandit: exploit a known strategy or "waste" some effort exploring alternatives in the hopes of better returns. The more efficient you are (exploitation), the higher the likelihood of catastrophic failure in weird edge cases.

CrowdStrike in this context is a NT kernel loadable module (a .sys file) which does syscall level interception and logs then to a separate process on the machine. It can also STOP syscalls from working if they are trying to connect out to other nodes and accessing files they shouldn't be (using some drunk ass heuristics).

What happened here was they pushed a new kernel driver out to every client without authorization to fix an issue with slowness and latency that was in the previous Falcon sensor product. They have a staging system which is supposed to give clients control over this but they pissed over everyone's staging and rules and just pushed this to production.

This has taken us out and we have 30 people currently doing recovery and DR. Most of our nodes are boot looping with blue screens which in the cloud is not something you can just hit F8 and remove the driver. We have to literally take each node down, attach the disk to a working node, delete the .sys file and bring it up. Either that or bring up a new node entirely from a snapshot.

This is fine but EC2 is rammed with people doing this now so it's taking forever. Storage latency is through the roof.

I fought for months to keep this shit out of production because of this reason. I am now busy but vindicated.

Edit: to all the people moaning about windows, we've had no problems with Windows. This is not a windows issue. This is a third party security vendor shitting in the kernel.

Interesting..

> We have to literally take each node down, attach the disk to a working node..

Probably the easiest solution for you is to go back in time to a previous scheduled snapshot, if you have that setup already.

I really want our cages, C7000's and VMware back at this point.

Is this how you got the inside scoop on the rollout fiasco?

I'm betting I have a good idea of one of the possible orgs you work for, since I used to work specifically with the largest 100 customers during my ~3yr stint in premium support

I used to have this crazy idea that fancy cloud vendors had competent management tools. Like maybe I could issue an API call to boot an existing instance from an alternate disk or HTTPS netboot URL. Or to insta-stop a VM and get block-level access to its disk via API, even if I had to pay for the instance while doing this.

And I’m not sure that it’s possible to do this sort of recovery at all without blowing away local SSD. There’s a “preview” feature for this on GCP, which seems to be barely supported, and I bet it adds massive latency to the process. Throwing away one’s local SSD on every single machine in a deployment sounds like a great way to cause potentially catastrophic resource usage when everything starts back up.

Hmm, I wonder if you’re even guaranteed to be able to get your instance back after stopping it.

WTF. Why can’t I have any means to access the boot disk of an instance, in a timely manner? Or any better means to recover an instance?

Is AWS any better?

...I remember that day pretty well. It was a busy day.

As for "throwing away local SSD", that only happens on AWS with instance store volumes which used to be called ephemeral volumes as the storage was directly attached to the host you were running on and if you did a stop/start of an ebs-backed instance, you were likely to get sent to a different host (vs. a restart API call, which would make an ACPI soft command and after a duration...I think it was 5 minutes, iirc, the hypervisor would kill the instance and restart it on the same host).

When the instance would get sent to a different host, it would get different instance storage and the old instance storage would be wiped from the previous host and you'd be provisioned new instance storage on the new host.

However, with EBS-volumes, those travel from host to host across stop/start cycles and they're attached via very low latency across the network from EBS servers and presented as a local block device to the instance. It's not quite as fast as local instance store, but it's fast enough for almost every use case if you get enough IOPS provisioned either through direct provisioning + correct instance size OR through a large enough drive + large enough instance to maximjze the connection to EBS (there's a table and stuff detailing IOPs, throughput, and instance size in the docs).

Also, support can detach the volume as well if the instance is stuck shutting down and doesn't get manually shut down by the API after a timeout.

None of this is by any means "ideal", but the complexity of these systems is immense and what they're capable of at the scale they operate is actually pretty impressive.

The key is...lots of the things you talk about are do-able at small scale, but when you add more and more operations and complexity to the tool stack on interacting with systems, you add a lot of back-end network overhead, which leads to extreme congestion, even in very high speed networks (it's an exponential scaling problem).

The "ideal" way to deal with these systems is to do regular interval backups off-host (ie. object/blob storage or NFS/NAS/similar) and then just blow away anything that breaks and do a quick restore to the new, fixed instance.

It's obviously easier said than done and most shops still on some level think about VMs/instances as pets, rather than cattle or have hurdles that make treating them as cattle much more challenging, but manual recovery in the cloud, in general, should just be avoided in favor of spinning up something new and re-deploying to it.

This is absurd. Every BMC I’ve ever used has an option to turn off the power immediately. Every low level hypervisor can do this, too. (Want a QEMU guest gone? Kill QEMU.). Why on Earth can’t public clouds do it?

The state machine for a cloud VM instance should have a concept where all of the resources for an instance are still held and being billed, but the instance is not running. And one should be able to quickly transition between this state and actually running, in both directions.

Also, there should be a way to force stop an instance that is already stopping.

The issue is far more nuanced than that. The systems are very complex and they're a hypervisor that has layers of applications and interfaces to allow scaling. In fact, the hosts all have BMCs (last I knew...but I know there were some who wanted to get rid of the BMC due to BMCs being unreliable, which is, yes, an issue when you deal with scale because BMCs are in fact unreliable. I've had to reset countless stuck BMCs and had some BMCs that were dead).

The hypervisor is certainly capable of killing an instance instantly, but the preferred method is an orderly shutdown. In the case of a reboot and a stop (and a terminate where the EBS volume is not also deleted on termination), it's preferred to avoid data corruption, so the hypervisor attempts an orderly shutdown, then after a timeout period, it will just kill it if the instance has not already shutdown in an orderly manner.

Furthermore, there's a lot more complexity to the problem than just "kill the guest". There are processes that manage the connection to the EBS backend that provides the interface for the EBS volume as well as apis and processes to manage network interfaces, firewall rules, monitoring, and a whole host of other things. If the monitoring process gets stuck, it may not properly detect an unhealthy host and external automated remediation may not take action. Additionally, that same monitoring is often responsible for individual instance health and recovery (ie. auto-recover) and if it's not functioning properly, it won't take remediation actions to kill the instance and start it up elsewhere. Furthermore, the hypervisor itself may not be properly responsive and a call from the API won't trigger a shutdown action. If the control plane and the data plane (in this case, that'd be the hypervisor/host) are not syncing/communicating (particularly on a stop or terminate), the API needs to ensure that the state machine is properly preserved and the instance is not running in two places at once. You can then "force" stop or "force" terminate and/or the control plane will update state in its database and the host will sync later. There is a possibility of data corruption or double send/receive data in a force case, which is why it's not preferred. Also, after the timeout (without the "force" flag), it will go ahead and mark it terminated/stopped and will sync later, the "force" just tells the control plane to do it immediately, likely because you're not concerned with data corruption on the EBS volume, which may be double-mounted if you start up again and the old one is not fully terminated.

>The state machine for a cloud VM instance should have a concept where all of the resources for an instance are still held and being billed, but the instance is not running. And one should be able to quickly transition between this state and actually running, in both directions.

It does have a concept where all resources are still held and billed, except CPU and Memory. That's what a reboot effectively does. Same with a stop (except you're not billed for compute usage and network usage will obviously be zero, but if you have an EIP, that would incur charges still). The transition between stop and running is also fast, the only delays incurred are via the control plane...either via capacity constraints causing issues placing an instance/VM or via the chosen host not communicating properly...but in most cases, it is generally a fast transition. I'm usually up and running in under 20 seconds when I start up an existing instance from a stopped state. There's also now a hibernate or sleep state that the instance can be put into if it's windows via the API where the instance acts just like a sleep/hibernate state of a regular Windows machine.

>Also, there should be a way to force stop an instance that is already stopping.

There is. I believe I referred to it in my initial response. It's a flag you can throw in the API/SDK/CLI/web console when you select "terminate" and "stop". If the stop/terminate command don't execute in a timely manner, you can call the same thing again with a "force" flag and tell the control plane to forcefully terminate, which marks the instance as terminated and will asynchronously try to rectify state when the hypervisor can execute commands. The control plane updates the state (though, sometimes it can get stuck and require remediation by someone with operator-level access) and is notified that you don't care about data integrity/orderly shutdown and will (once its updated the state in the control plane and regardless of the state of the data plane) mark it as "stopped" or "terminated". Then, you can either start again, which should kick you over to a different host (there are some exceptions) or you can launch a new instance if you terminated and attach an EBS volume (if you chose not to terminate the EBS volume on termination) and retrieve data (or use the data or whatever you were doing with that particular volume).

Almost all of that information is actually in the public docs. There was only a little bit of color about how the backend operates that I added for color. There's hundreds of programs that run to make sure the hypervisor and control plane are both in sync and able to manage resources and if just a few of them hang or are unable to communicate or the system runs out of resources (more of a problem on older, non-nitro hosts as that's a completely different architecture with completely different resource allocations), then the system can become partially functional...enough so that remediation automation won't step in or can't step in because other guests appear to be functioning normally. There's many different failure modes of varying degrees of "unhealthy" and many of them are undetectable or need manual remediation, but are statistically rare and by and large most hosts operate normally. On a normally operating host, forcing a shutdown/terminate works just fine and is fast. Even when some programs that are managing the host are not functioning properly, launch/terminate/stop/start/attach/detach all tend to continue to function (along with the "force" on detach, terminate, stop), even if one or two functions of the host are not functioning properly. It's also possible (and has happened several times) where a particular resource vector is not functioning properly, but the rest of the host is fine. In that case, the particular vector can be isolated and the rest of the host works just fine. It's literally these tiny little edge cases that happen maybe .5% of the time that cause things to move slower and at scale, a normal host with a normal BMC would have the same issues. Ie. I've had to clear stuck BMCs before on those hosts. Also, I've dealt with completely dead BMCs. When those states occur, if there's also a host problem, remediation can't go in and remedy host-level problems, which can lead to those control-plane delays as well as the need to call a "force".

Conclusion: it may SEEM like it should be super easy, but there's about a million different moving parts to cloud vendors and it's not just as simple as kill it with fire and vengeance (ie. quemu guest kill). BMCs and hypervisors do have an instant kill switch (and guest kill is used on the hypervisor as is a BMC power off in the right remediation circumstances), but you're assuming those things always work. BMCs fail. BMCs get stuck. You likely haven't had the issue because you're not dealing with enough scale. I've had to reset BMCs manually more times than I can count and I've also dealt with more than my fair share of dead ones. So, "power off immediately" does not always work, which means a disconnect occurs between the control plane and the data plane. There's also delays in remediation actions that automation takes to give enough time for things to respond to the given commands, which leads to additional wait time.

https://cloud.google.com/compute/docs/reference/rest/v1/inst...

AWS does indeed seem more enlightened:

https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_S...

I worked there for just shy of 7 years and dealt with capacity tangentially (knew a good chunk of their team for a while and had to interact with them frequently) across both teams I worked on (support and then inside the EC2 org).

Capacity, while their methodologies for expanding were, in my opinion, antiquated and unenlightened for a long time, were still rather effective. I'm pretty sure that's why they never updated their algorithm for increasing capacity to be more JIT. Now, they have a LOT more flexibility in capacity now that they have resource vectoring, because you no longer have hosts with fixed instance sizes for the entire host (homogenous). You now have the ability to fit everything like legos as long as it is the same family (ie. c4 with c4, m4 with m4, etc.) and there was additional work being done to have cross-family resource vectoring as well that was in-use.

Resource vectors took a LONG time for them to get in place and when they did, capacity problems basically went away.

The old way of doing it was if you wanted to have more capacity for, say, c4.xlarge, you'd either have to drop new capacity and build it out to where the entire host had ONLY c4.xlarge OR you would have to rebuild excess capacity within the c4 family in that zone (or even down to the datacenter-level) to be specifically built-out as c4.xlarge.

Resource vectors changed all that. DRAMATICALLY. Also, to reconfigure a hosts recipe now takes minutes, rather than rebuilding a host and needing hours. So, capacity is infinitely more fungible than it was when I started there.

Also, I think resource vectoring came on the scene around 2019 or so? I don't think it was there in 2018 when I went to work for EC2...but it was there for a few years before I quit...and I think it was in-use before the pandemic...so, 2019 sounds about right.

Prior to that, though, capacity was a much more serious issue and much more constrained on certain instance types.

In OCI we made a decision years ago that after 15 minutes from sending an ACPI shutdown signal, the instance should be hard powered off. We do the same for VM or BM. If you really want to, we take an optional parameter on the shutdown and reboot commands to bypass this and do an immediate hard power off.

So worst case scenario here, 15 minutes to get it shut down and be able to detach the boot volume to attach to another instance.

I’m just a free tier user but OCI is quite powerful. It feels a bit like KDE to me where sometimes it takes a while to find out where some option is, but I can always find it somewhere, and in the end it beats feeling limited by lack of options.

MS don't have testers any more. Where do you think CS learned their radically effective test-in-prod approach?

They shit all over our controls and went to production.

This says we don't control it and should not trust it. It is being removed.

Congratulations on actually fixing the root cause, as opposed to hand wringing and hoping they don't break you again. I'm expecting "oh noes, better keep it on anyway to be safe" to be the popular choice.

It's far easier for IT departments to just keep using it than it is to switch and managers will complain about "the cost of migrating" and "the time to evaluate and test a new solution" or "other products don't have feature X that we need" (even when they don't need that feature, but THINK they do).

Sure, but Windows shares some portion of the blame for allowing third-party security vendors to “shit in the kernel”.

Compare to macOS which has banned third-party kernel extensions on Apple Silicon. Things that once ran as kernel extensions, including CrowdStrike, now run in userspace as “system extensions”.

If a failure in a userspace service can crash the entire system, that's a bug.

linux, Solaris, BSD and macOS aren't without their flaws, but MSFT could have done a much better job with system design.

On macOS you can't delete or modify critical system files without both a root password and enough knowledge to disable multiple layers of hardware-enforced system integrity protection.

https://arstechnica.com/information-technology/2006/10/7998/

Honest question, I've seen comments in these various threads about people having similar issues (from a few months/weeks back) with kernel extension based deployments of CrowdStrike on Debian/Ubuntu systems.

I haven't seen anything similar regarding Mac OS, which no longer allows kernel extensions.

Is Mac OS not impacted by these kinds of issues with CrowdStrike's product, or have we just not heard about it due to the small scale?

Personally, it's a shared responsibility issue. MS should build a product that is "open to extension but closed for modification".

> they pissed over everyone's staging and rules and just pushed this to production.

I am guessing that act alone is going to create a massive liability for CrowdStrike over this issue. You've made other comments that your organization is actively removing CrowdStrike. I'm curious how this plays out. Did CrowdStrike just SolarWind themselves? Will we see their CISO/CTO/CEO do time? This is just the first part of this saga.

There is no shared responsibility. CrowdStrike pushed a broken driver out, then triggered the breakage, overriding customer requirement and configuration for staging. It is a faulty product with no viable security controls or testing.

And then you get midwits who blame Microsoft for allowing kernel access in the first place. Yes Apple deprecated kexts on macOS; that's a hell of a lot easier to do when you control the entire hardware ecosystem. Go ahead and switch to Apple then. If you want to build your own machines or pick your hardware vendor, guess what, people are going to need to write drivers, and they are probably going to want kernel mode, and the endpoint security people like CrowdStrike will want to get in there too because the threat is there.

There's no way for Microsoft or Linux for that matter to turn on a dime and deny kernel access to all the thousands upon thousands of drivers and system software running on billions of machines in billions of potential configurations. That requires completely reworking the system architecture.

This midwit spent the day creating value for my customers instead of spinning in my chair creating value for my cardiologist.

Microsoft could provide adequate system facilities so that customers can purchase products that do the job without having the ability to crash the system this way. They choose not to make those investments. Their customers pay the price by choosing Microsoft. It's a shared responsibility between the parties involved, inclduing the customers that selected this solution.

We all make bad decisions like this, but until customers start standing up for themselves with respect to Microsoft, they are going to continue to have these problems, and society is going to continue to pay the price all around.

We can and should do better as an industry. Making excuses for Microsoft and their customers doesn't get us there.

They aren’t perfect, but they are an improvement over what is available on Windows. Microsoft needs to get moving in this same direction.

It took a bit to figure out with some customers, but we provide optional VNC access to instances at OCI, and with VNC the trick seems to be to hit esc and then F8, at the right stage in the boot process. Timing seems to be the devil in the details there, though. Getting that timing right is frustrating. People seem to be developing a knack for it though.

I miss my AS/400.

This might be a decent place to recount the experience I had when interviewing for office security architect in 2003. my background is mainframe VM system design and large system risk management modeling which I had been doing since the late 80s at IBM, DEC, then Digital Switch and Bell Canada. My resume was pretty decent at the time. I don't like Python and tell VP/Eng's they have a problem when they can't identify benefits from JIRA/SCRUM, so I don't get a lot of job offers these days. Just a crusty greybeard bitching...

But anyway... so I'm up in Redmond and I have a decent couple of interviews with people and then the 3rd most senior dev in all of MSFT comes in and asks "how's your QA skills?" and I start to answer about how QA and Safety/Security/Risk Management are different things. QA is about insuring the code does what it's supposed to, software security, et al is about making sure the code doesn't do what it's not supposed to and the philosophic sticky wicket you enter when trying to prove a negative (worth a google deep dive if you're unfamiliar.) Dude cuts me off and says "meh. security is stupid. in a month, Bill will end this stupid security stand down and we'll get back to writing code and I need to put you somewhere and I figured QA is the right place."

When I hear that MSFT has systems that expose inadequate risk management abstractions, I think of the culture that promoted that guy to his senior position... I'm sure he was a capable engineer, but the culture in Redmond discounts the business benefits of risk management (to the point they outsource critical system infrastructure to third parties) because senior engineers don't want to be bothered to learn new tricks.

Culture eats strategy for breakfast, and MSFT has been fed on a cultural diet of junk food for almost half a century. At least from the perspective of doing business in the modern world.

"Although Channel Files end with the SYS extension, they are not kernel drivers."

https://www.crowdstrike.com/blog/technical-details-on-todays...

Which is horrible!

I'm a reporter with Bloomberg News covering cybersecurity. I'm trying to learn more about this Crowdstrike update potentially bypassing staging rules and would love to hear about your experience. Would you be open to a coversation?

I'm reachable by email at [email protected] or on Signal at JakeBleiberg.24. Here's my Bloomberg author page: https://www.bloomberg.com/authors/AWuCZUVX-Pc/jake-bleiberg.

Thank you.

Jake

And if all those recovery keys are stored in AD (as they usually are), and the Domain Controllers all had Crowdstrike on them...

So might save a few people, but I suspect not many.

Whoever only has AD-based Bitlocker encryption is straight up fucked. Man, and that on a Friday.

> Workaround Steps:

> Boot Windows into Safe Mode or the Windows Recovery Environment

> Navigate to the C:\Windows\System32\drivers\CrowdStrike directory

> Locate the file matching “C-00000291*.sys”, and delete it.

> Boot the host normally.

I guess more feasible and even neater to do it if you have network boot or similar.

The problem may be fixed but I can see some companies having a really shit weekend.

But someone probably fixed it, and the cars were able to go out on track for the first practice session.

If system administrator had too much free time, and configured every system to probe network on booting, and there is no encryption, it is possible to boot from a minimal Linux image with a script that automatically renames the driver and restarts.

The corporate version of the same approach uses Intel AMT (or however else it is called), but it is only available on licensed hardware from big suppliers.

Otherwise, you can distribute flash drives with the same auto-executing fix to everyone who is able to enter firmware setup, and boot from USB. If it's not available for security reasons, more manual work is required.

But what happens next? If Crowdstrike handled all the security measures, and there was no additional firewall rules, address checks, and so on, your network is now as open as it can be. I suppose certain groups have been celebrating, and uploading gigabytes of data from networks whose detection systems became severed.

Easier to just rebuild from the image. For every windows machine your company has. lol.

They're just bad at it.

I know, I'm dreaming.

I can't even open files larger than 500 lines without my whole system slowing to a crawl because of the insanely aggressive and slow "antivirus" bloatware the MDM forces on me.

So there wasn't any new kernel driver deployed, the existing kernel driver just doesn't fail gracefully.

[0]: https://x.com/brody_n77/status/1814185935476863321

What surprises me is if its a content update, and the code fell over when dealing with it - just basically bad release engineering isn't it not to cater for that in the first place? i.e. some tests in the pipeline before releasing the content update would've picked it up given it sounds like 100% failure rates.

They have enough client base that they can even run an A/B test on the whole binary level, but no.

Isn't this done as well with automatic updates of end user software or embedded systems and if not, why not?

Running Windows on bare-metal was always obviously very stupid. The consequences of such stupidity are just being felt now.

Premature deployment of Crowdstrike AGI disaster response plan.

We've all pushed bad updates but how was this not tested?

(also, great choice of name i must say)

and he's the former CTO of mcafee?

I mean, don't they do canary updates on CrowdStrike too? Every Windows admin has done this for the last 5+ years, test Windows updates on a small number of systems to see if they are stable. Why not do the same for 3rd party software?

on https://azure.status.microsoft/en-gb/status the message is currently:

> We have been made aware of an issue impacting Virtual Machines running Windows Client and Windows Server, running the CrowdStrike Falcon agent, which may encounter a bug check (BSOD) and get stuck in a restarting state.

https://ibb.co/Bc6n527

"62 minutes could bring your business down"

I guess they could bring all the businesses down much quicker.

https://www.crowdstrike.com/en-us/#teaser-79minutes-adversar...

(Repeating my comment because other story is duped)

"The issue has been identified, isolated and a fix has been deployed."

Maybe I'm misunderstanding what I read elsewhere, but is the machine not BSODing upon boot, prior to a Windows Update service being able to run? The "fix" I see on reddit is roughly:

Workaround Steps:

1. Boot Windows into Safe Mode or the Windows Recovery Environment

2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory

3. Locate the file matching “C-00000291*.sys”, and delete it.

I'm horrified at the thought of tens of thousands of novice Windows users digging through System32 to delete driver files; can someone set my mind at ease and assure me this will eventually be fixed in an automated fashion?

https://www.crowdstrike.com/blog/statement-on-windows-sensor...

It's like two hours of work with dnsmasq and a minimal Linux ISO. The only problem is that much of the work is not shareable between organisations; network structures differ, architectures may differ, partition layout may differ, the list of assets (and their MAC addresses) will differ.

Edit: + individual organisations won't be storing their BitLocker recovery keys in the same manner as each other either. You did back up the recovery keys when you enabled BitLocker, right? Modern cryptsetup(8) supports a BITLK extension for unlocking said volumes with a recovery key. Again, this can be scripted.

Because writing such a script (that mounts the filesystem and delete a file) under stress and time constraint is a great idea? That's a recipe for a worse disaster. The best solution, for now, is to go PC by PC manually. The sole reason the situation is as is was the lack of backstage testing.

Their lawyers certainly won't allow mentioning such dramatic (is "dramatic" appropriate here?) consequences.

Nope. Both my orgs (+2000 each) have sent out a Google doc to personal emails on using CMD Prompt to delete that file.

Anyone with technical experience is being drafted to get on calls and help people manually delete this file.

Please don't do this! It makes merging threads a pain. It's better to let us know at [email protected] and then we'll merge the threads so your comment shows up in the main one.

This is not a small startup with some SaaS, these guys are in most computers of too many huge companies. Not rolling out the updates to everyone at the same time seems just too obvious

Yeah let's throw an LLM at the C++ kernel driver and auto-push to prod

Microsoft needs to take control and forbid anyone and anything from running software with that kind of behavior.

https://www.nasdaq.com/market-activity/stocks/crwd/short-int...

But maybe this kind of thing can actually impart the lesson that loading your OS up with always-on, internet-connected agents that include kernel components in order to instrument every little thing any program does on the system is, uh, kinda risky.

But maybe not. I wonder if we'll just see companies flock to alternative vendors of the exact same type of product.

My org which does mission critical healthcare just deployed ZScaler on every computer which is now in the critical path of every computer starting up and then in the critical path of every network connection the computer makes. The risk of ZScaler being a central point of failure is not considered. But - the risk of failing the compliance checkbox it satisfies is paramount.

All over the place I'm seeing checkbox compliance being prioritised above actual real risks from how the compliance is implemented. Orgs are doing this because they are more scared of failing an audit than they are of the consequences failure of the underlying systems the audits are supposed to be protecting. So we need to hold regulatory bodies accountable as well - when they frame regulation such that organisations are cornered into this they get to be part of the culpability here too.

In corporate environments, IT staff struggle to contain these issues using antivirus software, firewalls, and proxies. These security measures often slow down PCs significantly, even on recent multi-core systems that should be responsive.

Microsoft is responsible for providing an operating system that is inherently insecure and vulnerable. They have prioritized user lock-in, dark patterns, and ease of use over security.

Apple has done a much better job with macOS in terms of security and performance.

The corporate world is now divided into two categories: 1. Software-savvy companies that run on Linux or BSD variants, occasionally providing macOS to their employees. These include companies like Google, Amazon, Netflix, and many others. 2. Companies that are not software-focused, as it's not their primary business. These organizations are left with Microsoft's offerings, paying for licenses and dealing with slow and insecure software.

The main advantage of Microsoft's products is the Office suite: Excel, Word and Powerpoint but even Word is actually mediocre.

EDIT: improve expression and fix errors:

I wonder why is it the case. These companies still have IT departments, someone has to manage these huge fleets of Windows machines. So nothing would prevent them from hiring Linux admins instead of Windows admins. What makes the management of these companies consider Windows to be the default choice?

But that aside — Excel is a single application. That one app doesn’t determine an entire Corporate IT strategy.

It actually does in some industries, but it is dumb that it does.

There are plenty of sufficient replacements for Excel if organizations are willing to work with other tools

1. Users are more comfortable running Windows and Office because it's Windows they likely used in school and on personal laptops.

2. This is the biggie: Microsoft's enterprise services for managing fleets of workstations are actually really good -- or at least a massive step up from the competition. Linux (and it's ilk) is much better for managing fleets of servers, but workstations require a whole different type of tooling. And once you have AD and it's ilk running and thus Windows administrators hired, it's often easier to run other services from Windows too, rather than having to spin up another cluster of management services.

Software focused businesses generally start out with engineers running macOS or Linux, so they wouldn't have Windows management services pre-provisioned. And that's why you generally see them utilising stuff like Okta or Google Workspace

Microsoft has a large, entrenched distribution network and market all over the world. It makes an uphill battle to create low-end programs for schools, universities, governments, SMBs.

Hence the phrase "no one was ever fired from buying Microsoft". It's too hard a battle to go against the flow.

[1] https://www.tomshardware.com/software/windows/microsofts-dra...

Still, all this is a red herring. Using Linux instead of Windows on workstations won't change anything, because it's not the OS that's the problem. A typical IT department is locked in a war on three fronts - defending against security threats, pushing back on unreasonable demands from the top, and fighting the company employees who want to do their jobs. Linux may or may not help against external attackers, but the fight against employees (which IT does both to fulfill mandates from the top and to minimize their own workload) requires tools for totalitarian control over computing devices.

Windows actually is better suited for that, because it's designed to constrain and control users. Linux is designed for the smart user to be able to do whatever they want, which includes working around stupid IT policies and corporate malware. So it shouldn't be surprising corporate IT favors Windows workstations too - it puts IT at an advantage over the users, and minimizes IT workload.

This just tells me you don't know linux. Linux can be much more easily hardened and restricted than windows. It's trivial to make it so that a user can only install whitelisted software from private repos.

This is also straightforward on Windows with AD-managed Group Policies.

A lot actually don’t, in any meaningful sense. My partner’s company has a skeleton IT staff with all support requests being sent offshore. An issue with your laptop? A new one gets dispatched from ??? and mailed to you, you mail the old one back, presumably to get wiped and redispatched to the new person that has a problem.

If linux had software anywhere close to the amount that windows has, it would have experienced the same issues too. After all it is not just about running a server and tinkering with config files. It is about ability to manage the devices, rolling out updates and so on.

And Linux is not even a particularly hardened OS. If we could take some resources from VC smoke and mirrors and dedicate them to securing our critical infrastructure we could have airports and hospitals running on some safety-critical microkernel OS with tailored software.

Technical arguments, logic, and sense do not contribute much to purchasing decisions in the corporate world.

With NixOS, you can upgrade without fear, as you can always roll back to a previous version of your system. Regular Linux distributions, macOS, and Windows make me very nervous because that is not the case.

NixOS isn't immutable, things aren't mounted read only. AFAIK, it can't be setup that way.

> With NixOS, you can upgrade without fear, as you can always roll back to a previous version of your system. Regular Linux distributions, macOS, and Windows make me very nervous because that is not the case.

Because you can't roll back to a previous backup?

Backups could be an option, but it is much better to have a system where two computers are guaranteed to be running the exact same software if configuration hashes are the same.

In other OSes, the state of your system could depend on previous actions.

I'm personally only really nervous when updating Linux distributions. Besides security updates it usually hardly matters or is noticeable on macOS/Windows (well besides the random UX changes..).

This is also true of most regular Linux setups. Except that in those, you can only choose the kernel. Hence, if you have broken other parts of your configuration, your system might not be bootable. So the safety net is much thinner.

Maybe things have gotten better, but I think that's why people use Mac. It's POSIX but without having to jump through arcane hoops.

The problem with the linux desktop was usually that most hardware companies were either not spending any time/effort on non-windows drivers/compatibility or when they did it was a tiny fraction of the effort that went into working around bugs in the windows driver API's.

Today with the failure of windows in both the mobile and industrial control space we now see vendors actually giving a damn about the quality of their Linux drivers.

Today the main factor keeping the enterprise marked locked on windows is the fat clients written around the turn of the millennium, and that's as much a problem for mac adaptation as it is Linux adaptation.

The macs are slick well designed devices that speaks to a huge segment of the consumer market so will eventually find the way into the high cost niches where no specific dependency on legacy software exists but they are too expensive and inflexible to replace all of the wintel system so for Microsoft and it's partners to have their license to screw over the enterprise sector revoked Linux(or FreeBSD) will have to play a role too.

It was an HP from Costco, not something special sold with Linux. My wireless worked, dual monitors just worked, even the fingerprint reader that I never use. I remember sitting there thinking "I didn't have to fight anything?" Hopefully that becomes the norm, maybe it is - I haven't needed a new laptop yet.

Because you just want stuff to work and couldn't care less about the ideology part?

Also no feature parity (it's not about Windows being "better" than Linux or the other way around, none of that matters) there are not out of the box solutions to replace some of the stuff enterprise IT relies in Windows/etc. which would mean they'd have to hire expensive vendors to recreate/migrate their workflows. The costs of figuring out how to run all of your legacy Windows software, retraining staff etc. etc. would be very significant. Why spend so much money with no clear benefits?

To be fair I'm not sure how Apple figures into this. They don't really cater to the enterprise market at al..

I think the current outage undercuts this premise.

Yes, but that's not because of Windows itself (which is fast and secure out of the box) but because of an decades-old "security product" culture that insists on adding negative-value garbage like Crowdstrike and various anti-virus systems on the critical path, killing performance and harming real security.

It's a hard problem. No matter how good Windows itself gets and no matter how bad these "security products" become, Windows administrators are stuck in the same system of crappy incentives.

Decades of myth and superstition demand they perform rituals and make incantations they know harm system security, but they do them anyway, because fear and tradition.

It's no wonder that they see Linux and macOS as a way out. It's not that they're any better -- but they're different, and the difference gives IT people air cover for escaping from this suffocating "you must add security products" culture.

That's a really bold claim. I'd say Windows comes with a lot of unsafe defaults OOB.

> Yes, but that's not because of Windows itself

Come on. There’s a reason Windows users all want to install crappy security products: they’ve been routinely having their files encrypted and held for ransom for the last decade.

But most of Windows falling to this is that it’s what people use. The only platform that is somewhat actually protected against attacks is the iPhone - the Mac can easily be ransomwared it’s just the market is so small nobody bothers attacking it; no ROI.

I have a similar disdain for security bloatware with questionable value, but one actually effective corporate IT strategy is using one of those tools to operate a whitelist of safe software, with centralized updates

In Windows land it's pretty much expected that you go to random websites, download random executables, ignore the "make changes to your computer?" warnings and pretty much give the exe full permission to do anything. It's very much been the standard software install workflow for decades now on Windows.

In the Linux/BSD world, while you can do the above, people generally don't. Generally, they stick to trusted software sources with centralized updates, like your second point. In this case I don't think it's a matter of capability, both Windows and Unix-land is capable of what you're suggesting.

I think phishing is generally much less effective in Max/Linux/BSD world because of this.

And no one would pipe a script downloaded with wget/curl directly into bash.

And nobody would copy a script from a code-formatted block on a page, paste it directly into their terminal and then run it.

Im not going to go so far as to claim that these behaviors are as common as installing software on Windows, but they are still definitely common, and all could lead to the same kinds of bad things happening.

I think though that the culture of robust repositories and package managers is MUCH more prominent on Mac/iOS/Linux/FreeBSD. It's coming to Windows too with the new(er) Windows store stuff, so hopefully people don't become too resistant to that.

Basically any userspace program can read your .aws, .ssh, .kube, etc... The user based security model desktops have is the real issue.

Compare that with Android and iOS for instance. No one needs anti-virus bloatware, just because apps are curated and isolated by default.

It comes from the 1990s and early 2000s. Back then, Windows was a laughingstock from a security point of view (for instance, at one point connecting a newly installed Windows computer to the network was enough for it to be automatically invaded). Both Windows and Linux have become more secure since then.

> Basically any userspace program can read your .aws, .ssh, .kube, etc... The user based security model desktops have is the real issue. Compare that with Android and iOS for instance. No one needs anti-virus bloatware, just because apps are curated and isolated by default.

Things are getting better now, with things like flatpak getting more popular. For instance, the closed-source games running within the Steam flatpak won't have any access to my ~/.aws or ~/.ssh or ~/.kube or etc.

Yes, average Windows users are significantly less tech literate due to obvious reasons and there are way more of them. This create a very lucrative market.

How is desktop Linux somehow inherently particularly more secure than Windows?

Some of the “compliance managers as a service” push you in this direction as well.

If you’re a startup it also means that your core people are now sitting in multiple cycles of IT review with their IT staff filling out spreadsheet after spreadsheet of “Do you encrypt data in transit?”

SMB 1.0 is enabled, non admin users have powershell access, defender can be disabled with a single command, user is admin by default, passwords can be reset via booting to a bootable media device and then swapping its CLI to c:

there are so many basic insecurities out of the box in windows.

I think what you're really saying is that a Windows system is secure until you apply power to the computer.

Disagree. At least in the context of business networks.

My favorite example is the SMB service, which is enabled by default.

In the Linux world, people preach:

- disabling SSH unless necessary

- use at least public key-based auth

- better both public key and password

- don't allow root login

In Windows, the SMB service:

- is enabled by default

- allows command execution as local admin via PsExec, so it's essentially like SSH except done poorly

- is only password-based

- doesn't even support MFA

- is not even encrypted by default

It's a huge issue why everyone gets encrypted by ransomware.

I always recommend disabling it using the Windows firewall unless it is actually used, and if it is necessary define a whitelist of address ranges, but apparently it is too hard to figure out who needs access to what, and much easier to deploy products like Crowdstrike which admittedly strongly mitigate the issue.

The next thing is that Windows still allows the NTLM authentication protocol by default (now finally about to be deprecated), which is a laughably bad authentication protocol. If you manage to steal the hash of the local admin on one machine, you can simply use it to authenticate to the next machine. Before LAPS gained traction, the local admin account password was the same on all machines in basically every organization. NT hashes are neither salted nor do they have a cost factor.

I could go on, but Microsoft made some very questionable security decisions that still haunt them to this day because of their strong commitment to backwards compatibility.

All I did was challenge the statement that Windows is secure OOB.

Do not underestimate corporate IT's ability to slow down Macs with endpoint security software.

I used to run a C++ shop, writing heavy-duty image processing pipeline software.

It did a lot, and it needed to do it in realtime, so we were constantly busting our asses to profile and optimize the software.

Our IT department insisted that we install 'orrible, 'orrible Java-based sneakware onto all of our machines, including the ones we were profiling.

We ended up having "rogue" machines, that would have gotten us in trouble, if IT found out (and I learned that senior management will always side with IT, regardless of whether or not that makes sense. It resulted in the IT department acting like that little sneak that sticks his tongue out at you, while hiding behind Sister Mary Elephant's habit).

But, to give them credit, they did have a tough job, and the risks were very real. Many baddies would have been thrilled to get their claws on our software.

As I looked through I killed sophos. Suddenly speeds shot up above 7gbit. A few seconds later they dropped back down, sophos has retured.

A "while (true) pkill sophos" later and the malware was sedated.

Having proved it wasn't a network problem I left it with the engineer to determine the best long term solution.

We were in need of an MDM to help staff (non-techs) with their Mac books. I haven't noticed any issues, nor have two of my staff who are trialling it. What's been your main gripe?

I'm a Dev but also manage the It team of one sys admin and haven't noticed any performance hits. Yet anyway, but it's only been two weeks.

I haven't had the same experience but that's mostly because I'm a Dev, implementing it for other developers.

It sounds less like a product problem vs. configuration problem caused by your IT teams.