News for hackers. Stuff that matters.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by nimbius on Friday July 19, 2024 @01:02PM (#41008550)
i work for a diesel truck maintenance and repair shop and its been hell on earth this morning.

- our IT wizard says the fixes wont work on lathes/CNC systems. we may need to ship the controllers back to the manufacturer in Wisconsin.

- AC is still not running. sent the apprentice to get fans from the shop floor.

- building security alarms are still blaring, need to get a ladder to clip the horns and sirens on the outside of the building. still cant disarm anything.

- still no phones. IT guy has set up two "emergency" phones...one is a literal rotary phone. stresses we still cannot call 911 or other offices. fire sprinklers will work, but no fire department will respond.

- no email, no accounting, nothing. I am going to the bank after this to pick up cash so i can make payday for 14 shop technicians. was warned the bank likely would either not have enough, or would not be able to process the account (if they open at all today.)

Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by dan_quixote on Friday July 19, 2024 @01:19PM (#41008745)
> our IT wizard says the fixes wont work on lathes/CNC systems

Why, whY, WHY...are these things connected to the internet?!

Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by jdietrich on Friday July 19, 2024 @02:30PM (#41009555)
Remote monitoring, analytics and diagnostics have a significant impact on uptime, utilisation and profitability. You're thinking in terms of a single machine, but the managers of machine shops are thinking in terms of a complex process across many machines and often across many sites. Some of that functionality could be delivered using an airgapped network, but a lot of important features essentially require an internet connection.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by acuozzo on Saturday July 20, 2024 @12:33PM (#41017675)
An embedded controller can deliver all of this information over a serial line to a central hub.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by commandlinefan on Friday July 19, 2024 @01:54PM (#41009159)
> WHY...are these things connected to the internet

Because the manufacturer makes sure they don't start up if they're not. Otherwise how else would they be able to spy on you?

Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by skrebbel on Friday July 19, 2024 @02:15PM (#41009370)
Source?
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by betaby on Friday July 19, 2024 @03:03PM (#41009898)
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by skrebbel on Friday July 19, 2024 @03:12PM (#41009992)
That’s not a lathe nor a CNC system. Again, which CNC manufacturers are installing windows + crowdstrike on their machines just so they can spy on their customers? You’re all just spreading conjecture. This attitude isn’t at all as widespread (nor profitable) in low(ish) volume B2B hardware sectors.

These industries have terrible track records wrt security and even software robustness, but they don’t routinely spy on their customers for weird marketing reasons. If there’s remote connectivity it’s for real reasons (eg remote maintenance, updates etc).

The suggestion that CNC machines run internet connected windows+crowdstrike just so the manufacturer can spy on their customers strikes me as pretty ridiculous and your garage door story doesn’t really relate. Much more likely that they do it for (possibly bad) non-malicious reasons.

Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by more_corn on Friday July 19, 2024 @01:57PM (#41009180)
And charge you
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by foobarian on Friday July 19, 2024 @01:31PM (#41008883)
And why are they running windows? And why are they running Crowdstrike? WTF
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by consp on Friday July 19, 2024 @01:40PM (#41008980)
If they are offline it should not matter which OS they run, maybe a RTOS for the control software but anything goes for the ux.

If they are online, well...

Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by MangoCoffee on Friday July 19, 2024 @02:22PM (#41009450)
Not sure what the OS has to do with CrowdStrike's fuckup. CrowdStrike also runs on Linux and macOS.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by gradyfps on Friday July 19, 2024 @02:32PM (#41009598)
The boot loop / BSOD issues are Windows specific.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by lend000 on Friday July 19, 2024 @02:30PM (#41009559)
And yet, of course it happens to Windows.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by skrebbel on Friday July 19, 2024 @03:13PM (#41010013)
This thread has multiple anecdotes of the same happening on their Linux version earlier.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by sameerds on Friday July 19, 2024 @11:12PM (#41013738)
It's probably the remote computer that's running Windows and currently affected by Crowdstrike.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by phkahler on Friday July 19, 2024 @02:25PM (#41009494)
>> Why, whY, WHY...are these things connected to the internet?!

Because SCADA systems. It's worthwhile to have an overview of an entire plant up in the main office. You can easily see what's running, what's not and what's got problems that need fixed.

Now for a small shop running jobs individually, they should definitely NOT be connected to the internet or even the LAN. But hey, some people think a thermostat needs to be on the network so there's that...

Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by octacat on Friday July 19, 2024 @02:08PM (#41009310)
some of them even have GPS. To prevent selling to sanctioned countries or reselling in general.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by HarryHirsch on Friday July 19, 2024 @02:13PM (#41009348)
Why, whY, WHY...are these things connected to the internet?!

It's so that the support engineer at the manufacturer can log in to troubleshoot. And then company IT support sprinkles a layer of antivirus on top. That's how we got here.

Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by o34jto432jt on Friday July 19, 2024 @02:28PM (#41009533)
So the manufacturer can sell you a "cloud connected service plan" where they change the font once every six months.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by rustcleaner on Friday July 19, 2024 @03:05PM (#41009918)
For our entertainment in times like this, of course!

grabs a bucket of popcorn and takes cover

Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by mschuster91 on Friday July 19, 2024 @01:50PM (#41009104)
CNC literally stands for "Computer numerical control". They're like the OG 3D printers, they just work subtractively than additive, and at much much much better precision.

You absolutely need computers to control them and loading up models via USB sticks becomes annoying rather fast, so naturally the control computers are network connected.

Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by swatcoder on Friday July 19, 2024 @02:00PM (#41009231)
"Network connected" or "conveniently programmable" !== "Internet connected"

It was a rhetorical question. I'm sure the GP knows what the machines are and why they might need some kind of convenient data supply.

Both manufacturers and on-site IT teams have simply gotten cavalier about internet connectivity, network isolation, automatic updates, etc -- convincing themselves that the catastrophic risks that come along with these processes will either not happen to them or will only happen when someone else can be blamed.

Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by willcipriano on Saturday July 20, 2024 @09:25AM (#41016326)
Tinfoil hat: Government might want to track/limit/<remotely brick> CNC machine usage someday to say prevent weapons manufacture and encourages this behavior in a similar manner to the way it encourages social media platforms to censor speech. Some of the really advanced CNC machines have GPS in them and won't work in "bad" countries.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by AzzyHN on Friday July 19, 2024 @01:08PM (#41008614)
Why are these things running Windows!?
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by poopsmithe on Friday July 19, 2024 @01:17PM (#41008719)
OS is irrelevant in this case, and CrowdStrike deserves all the blame. They literally brought down Linux systems earlier this year. https://news.ycombinator.com/item?id=41005936
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by seba_dos1 on Friday July 19, 2024 @01:27PM (#41008840)
It's still a valid question, just not directly related to the crashes.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by Rinzler89 on Friday July 19, 2024 @01:31PM (#41008884)
Because almost everything industrial runs windows because that's what the devs of those companies were most familiar with since MS-DOS days and evolved organically over time to modern versions of Windows due to great backwards compatibility and platform familiarity.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by consteval on Friday July 19, 2024 @04:28PM (#41010788)
Right but typically embedded systems run Linux, because while Windows has great compatibility on x86 it's virtually worthless outside of that.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by Rinzler89 on Friday July 19, 2024 @04:39PM (#41010915)
Those aren't embedded systems though, but mini PC computers. And embedded systems often run bare metal C code, not always Linux, especially for spindle/servo control where they get their commands from that PC.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by Redoubts on Friday July 19, 2024 @01:14PM (#41008689)
A lot of industrial machinery is just $x00,000 of equipment strapped to a windows pc. Hell, a lot of it is strapped to a version long EOL
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by porkbeer on Friday July 19, 2024 @01:19PM (#41008740)
Can confirm this is the norm in machine shops. I encounter systems running dos, 3.1, 95, 2k and mostly XP constantly. I rather prefer the old dos systems of the obsolete stuff. Less variables. It is easier and more reliable to freeze the tech in time than it is to manage updates.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by anon_cow1111 on Friday July 19, 2024 @01:27PM (#41008839)
My last CNC job was just a 98 pc that dropped into dos to load programs, this must have been right around when win10 came out. Sneakernet and floppies made it secure enough, but the main network where all the orders were handled was... terminal based.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by delta_p_delta_x on Friday July 19, 2024 @02:19PM (#41009417)
There are a lot of things running Windows because it's pretty straightforward to write a user-mode driver to interact with custom hardware compared to Linux, where every driver needs to be in the kernel and built with the kernel. Yes, there's DKMS, but it's still more of a faff than the relatively plug-and-play mechanism that Windows offers, especially since Vista.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by adastra22 on Friday July 19, 2024 @01:11PM (#41008661)
Almost every system you interact with in the world has some critical thing in its innards running windows.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by jordanb on Friday July 19, 2024 @01:49PM (#41009084)
I like the idea that technology is so unreliable in star trek because the computers are all centuries of software accretion with Windows way down the stack somewhere.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by adastra22 on Friday July 19, 2024 @02:05PM (#41009284)
The late great Vernor Vinge explored this in A Fire Upon The Deep. One of the characters is/was in a former life a programmer-archeologist. The idea being that so many thousands of years in the future every relevant program has already been written, so his job was to comb the archives for the right mix of codes and integrate them, rather than right something new.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by Freak_NL on Friday July 19, 2024 @03:35PM (#41010231)
“So we've got this CNC controller written in Rust from 2036, and, ah, here is a GUI for something like that written in late 90's Visual Basic 6… Just combine those and…”

“So uhm, you do know what you are doing, right?”

“Sir! I am a programmer-acheologist! Oh this is fascinating… Hold on, I must unearth and preserve this beauty of a BAT-file before we can go any further.”

Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by lapink on Friday July 19, 2024 @01:20PM (#41008758)
Most stuff is fine in France.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by slenk on Friday July 19, 2024 @01:31PM (#41008879)
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by dboreham on Friday July 19, 2024 @01:31PM (#41008891)
No Crowdstrike salespeople in France?
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by rvnx on Friday July 19, 2024 @02:01PM (#41009239)
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by surfingdino on Friday July 19, 2024 @01:57PM (#41009179)
Because Microsoft is giving away licenses to unis, esp in developing countries. IT jobs are seen there as a way to earn a good living and you get hordes of people who know nothing but Windows. That's how you get into the situation where most of the toolchains for embedded systems run on Windows, software for embedded systems is written on and for Windows, and so on. And then, one botched update fucks up everything.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by acedTrex on Friday July 19, 2024 @01:19PM (#41008742)
The automation world runs on windows
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by pphysch on Friday July 19, 2024 @01:14PM (#41008688)
More specifically why are they running extra endpoint management software that receives automatic updates from the Internet...

This is basically the IoT apocalypse scenario (AC is down??) but ironically not affecting many IoT devices, I assume.

Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by rolph on Friday July 19, 2024 @01:09PM (#41008625)
props to your IT guy for setting up in house phones. I believe these should be kept and expanded upon.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by a0123 on Friday July 19, 2024 @01:17PM (#41008724)
Feels like there is some sort of lesson we should learn as a society.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by curious_cat_163 on Friday July 19, 2024 @01:57PM (#41009194)
Not trust Microsoft to be the operating system of the modern life?
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by surfingdino on Friday July 19, 2024 @01:58PM (#41009208)
Yes, monocultures are disasters in the making.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by thepasswordis on Friday July 19, 2024 @01:17PM (#41008720)
Your IT wizard is probably wrong. There is a fix that involves booting into safe mode and deleting a file.

Unless you have an encrypted file system this should be a relatively trivial fix.

Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by John23832 on Friday July 19, 2024 @01:22PM (#41008771)
Machinery shipped to users usually do not allow for the users of the machinery to "boot into safemode". Thank John Deere and the anti-"Right to Repair" crowd for that.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by knd775 on Friday July 19, 2024 @01:55PM (#41009169)
I doubt it was shipped from the factory with CrowdStrike, and if they had enough access to install it, they have enough access to fix it.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by throwaway48476 on Friday July 19, 2024 @02:20PM (#41009422)
These things are "cost optimized" and don't feature the kind of remote management iDRAC/openBMC/piKVM that would allow it to be remotely fixed. Embedded windows connected to the internet is super ***.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by John23832 on Friday July 19, 2024 @04:56PM (#41011107)
You can doubt, but you literally don’t know.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by rahkiin on Friday July 19, 2024 @01:20PM (#41008754)
CNCs might not allow direct Windows access for end-users and require on-premise support from the manufacturer. Our cnc can be remotely serviced… if Windows boots.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by mardifoufs on Friday July 19, 2024 @01:48PM (#41009082)
So the CNC manufacturers pay for crowdstrike licenses? That's crazy
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by stackskipton on Friday July 19, 2024 @01:54PM (#41009151)
Yes because compliance requirements say “EDR must be installed on all machines”.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by HeWhoLurksLate on Saturday July 20, 2024 @01:45AM (#41014363)
Out of curiosity, what set of requirements would that be for?
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by rahkiin on Saturday July 20, 2024 @08:02AM (#41015863)
I’ve seen comments mention banking, or privacy references maybe when handling SSN and birthdates together (airports, hospitals?).
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by stackskipton on Saturday July 20, 2024 @03:21PM (#41018969)
SOC2, PCI, FedRAMP, cyber insurance. Just about any cybersecurity related compliance will have "All machines must have EDR."
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by sudosysgen on Friday July 19, 2024 @01:53PM (#41009148)
They probably have to due to CrowdStrike lobbying and fear-mongering requirements for their kind of software into export-controlled hardware.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by thepasswordis on Friday July 19, 2024 @01:24PM (#41008806)
If you’ve got physical access to the machine it’s your machine. All you need is a USB port.

I’d expect that the manufacturer puts out their own fix which basically copies crowdstrikes suggestion. I’d even suspect it by the end of the day today.

The fix is really simple, and luckily also very simple to automate. It’s going to be a lot of running around for IT staff (if deputized helpers!) but this should all be over by the weekend.

Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by freedomben on Friday July 19, 2024 @01:44PM (#41009032)
> If you’ve got physical access to the machine it’s your machine. All you need is a USB port.

You're a few years out of date here. Physical access is not the end like it used to be. We live in an era of hardware-backed anti-tamper and signed loaders/kernels.

If you have a way around it, I suggest you start reaching out these companies because you could make a lot of money.

Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by thepasswordis on Friday July 19, 2024 @03:36PM (#41010247)
Fair enough and I've been out of IT for a while. I wish I was still in it though, I'd love to be working on this!
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by kichimi on Friday July 19, 2024 @01:44PM (#41009031)
No it really is not. If you have a service contract, you do not touch it.
Re:CrowdStrike fixes start at "reboot up to 15 times", get m... by rahkiin on Friday July 19, 2024 @03:59PM (#41010485)
Nobody is going to try physically tampering with the HMI attached to their 50k$+ machine when you have a support contract indeed