Re:Maersk reinstalled thousands of servers and workstations ...
by scribu on Sunday January 28, 2018 @04:31PM
(#16253653)
For me, the most surprising thing is that Maersk was able to carry on during those 10 days without any IT:
> "We only had a 20% drop in volume, so we managed 80% of that volume manually. [...] Customers were great contributors to overcoming that."
That makes me think that either a) IT isn’t quite as essential as we software people like to think, or b) the full potential of automation hasn’t been achieved yet.
Re:Maersk reinstalled thousands of servers and workstations ...
by Animats on Monday January 29, 2018 @01:09AM
(#16255637)
It was much worse than Maersk's PR indicated. Their big automated ports, LA, NY, and Rotterdam, were down for most of a week. Maersk operates in about 160 ports, and about 17 were affected by the computer attack.
The Port Authority of New York and New Jersey has a message page for announcements to truckers. They're very real-world: "Gate 6 is closed due to flooding from a broken water main" kind of info. They were saying Maersk was totally down, then only handling inbound refrigerated containers, then inbound only. Later announcements mentioned manual processing, few gates open, long wait times, and suggested delaying shipments. It was about two weeks before they were fully back up and accepting everything, including empties.
LA was down for four days, but Maersk also has a terminal at Long Beach which was unaffected, so they diverted one container ship there. They weren't accepting online orders for a week or two, the tracking system was down, email was down, and they were trying to operate by voice phone only.
Many customers had containers stuck in the container stacks at the port.
Rotterdam's robotic container handling was down for a while, but details are scanty. Most of the problems were apparently with the truck-side systems, the ones which get the right container onto the right truck.
Re:Maersk reinstalled thousands of servers and workstations ...
by jsiepkes on Sunday January 28, 2018 @05:35PM
(#16253994)
They just started loading containers onto ships based on the best available information. Information ports had, customers, etc. A lot of containers ended up in the wrong place or simply went missing. It became a mess that will take them at least a year to fully cleanup.
Re:Maersk reinstalled thousands of servers and workstations ...
by erk__ on Sunday January 28, 2018 @05:07PM
(#16253870)
They still had a loss on $200-300 million, after everything. When it happend there was stories about how workers switched to using WhatsApp on their own phone to communicate.
Re:Maersk reinstalled thousands of servers and workstations ...
by malux85 on Sunday January 28, 2018 @05:09PM
(#16253878)
Some of it must have been sympathy by the customers too - "We got hacked and and to re-install thousands of machines" as a customer I'd let some late shipping slide. But could the company operate at a permanent performance penalty - i.e. without IT systems. They probably couldn't stay competitive
Re:Maersk reinstalled thousands of servers and workstations ...
by tjoff on Sunday January 28, 2018 @04:49PM
(#16253772)
I guess there are some factors that helps in this particular case. These are large container ships, the schedule is known very long in advance (the trips themselves take a long time too). After the initial hit the ships would have plenty of time to prepare before hitting port. Not trying to diminish the effort though! It is hard to imagine how they managed to hold everything together.
As mentioned in the embedded clip, it has the potential to get much worse in the future where even the boats are completely autonomous.
Also mentioned was that 90% of the orders came from internet, so 10% is not? Surprised me a little. Got the feeling that they have lots of manual paperwork even for internet orders, which probably helped a lot.
Re:Maersk reinstalled thousands of servers and workstations ...
by andy_ppp on Sunday January 28, 2018 @05:25PM
(#16253948)
From what I understand shipping is still an industry that operates between a few connected people. You still can barely go online and book a set of containers onto a ship, you call someone you have a relationship with. I assume that’s a big part of it.
Re:Maersk reinstalled thousands of servers and workstations ...
by tinymollusk on Sunday January 28, 2018 @08:26PM
(#16254719)
They had years of organization using IT that built up a lot of inertia. As another commenter said, 10 days is probably well below the threshold of their typical shipment turnover.
My guess is they'd quickly suffer potentially company-ending losses if they were to stop using modern IT altogether.
Re:Maersk reinstalled thousands of servers and workstations ...
by bkor on Monday January 29, 2018 @05:35AM
(#16256677)
> For me, the most surprising thing is that Maersk was able to carry on
It takes a while for one vessel to reach the next port/terminal. Once it left the previous terminal you'll get a stowage layout. Basically showing all the containers on a vessel and where they're supposed to unload/discharge.
You can use that file to tell the next terminal to unload whatever the file mentions. Then load whatever customers handed into the terminal and have the terminal handle the new stowage layout.
Thanks to loads of practical problems terminals are pretty capable to handle loads of things for you.
Obviously if the terminal itself is also affected it's a way bigger problem.
Re:Maersk reinstalled thousands of servers and workstations ...
by lfam on Sunday January 28, 2018 @06:06PM
(#16254143)
The article says that Maersk is taking the opposite approach. Quoting the CEO, Jim Hagemann Snabe:
"And this was a wake-up call to become not just good —we actually have a plan to come in a situation where our ability to manage cyber-security becomes a competitive advantage."
Re:Maersk reinstalled thousands of servers and workstations ...
by canada_dry on Sunday January 28, 2018 @06:21PM
(#16254213)
Being a (former/retired) IT Exec this kinda thing used to keep me up at night!
Sadly, most C-level folks don't understand that even gold-standard cyber security is far from perfect, and IT can't do much against dumb, careless users.
Re:Maersk reinstalled thousands of servers and workstations ...
by cm2187 on Monday January 29, 2018 @03:52AM
(#16256267)
But in this case they had not patched their windows machines for a vulnerability that Microsoft fixed months before. It was really a case of dumb, careless IT.
Re:Maersk reinstalled thousands of servers and workstations ...
by toomuchtodo on Sunday January 28, 2018 @09:59PM
(#16255040)
Disaster recovery/business continuity is like insurance: you grumble about paying it (or don’t pay it at all) and then you’re happy you did pay for it when disaster strikes.
Re:Maersk reinstalled thousands of servers and workstations ...
by Simulacra on Sunday January 28, 2018 @03:29PM
(#16253286)
Wow, that is a monumental effort. We can't even get a critical software patch in less than three months. I would be very curious to see what their new infrastructure defense plan looks like now compared to the old
Re:Maersk reinstalled thousands of servers and workstations ...
by lmm on Monday January 29, 2018 @05:51AM
(#16256730)
Sure, because most people using e.g. Apache run Linux, but it doesn't make sense to count it as a "linux bug" any more than it makes sense to count e.g. Ruby bugs as Linux bugs, or .net CLR bugs as Windows bugs.
Re:Maersk reinstalled thousands of servers and workstations ...
by kelnos on Monday January 29, 2018 @01:52AM
(#16255790)
There's nothing wrong with security through obscurity, as long as it is merely one part in a defense-in-depth strategy, and can help you on occasion when other defenses fail in unexpected ways.
> "We only had a 20% drop in volume, so we managed 80% of that volume manually. [...] Customers were great contributors to overcoming that."
That makes me think that either a) IT isn’t quite as essential as we software people like to think, or b) the full potential of automation hasn’t been achieved yet.
The Port Authority of New York and New Jersey has a message page for announcements to truckers. They're very real-world: "Gate 6 is closed due to flooding from a broken water main" kind of info. They were saying Maersk was totally down, then only handling inbound refrigerated containers, then inbound only. Later announcements mentioned manual processing, few gates open, long wait times, and suggested delaying shipments. It was about two weeks before they were fully back up and accepting everything, including empties.
LA was down for four days, but Maersk also has a terminal at Long Beach which was unaffected, so they diverted one container ship there. They weren't accepting online orders for a week or two, the tracking system was down, email was down, and they were trying to operate by voice phone only. Many customers had containers stuck in the container stacks at the port.
Rotterdam's robotic container handling was down for a while, but details are scanty. Most of the problems were apparently with the truck-side systems, the ones which get the right container onto the right truck.
This was a big deal.
As mentioned in the embedded clip, it has the potential to get much worse in the future where even the boats are completely autonomous.
Also mentioned was that 90% of the orders came from internet, so 10% is not? Surprised me a little. Got the feeling that they have lots of manual paperwork even for internet orders, which probably helped a lot.
My guess is they'd quickly suffer potentially company-ending losses if they were to stop using modern IT altogether.
It takes a while for one vessel to reach the next port/terminal. Once it left the previous terminal you'll get a stowage layout. Basically showing all the containers on a vessel and where they're supposed to unload/discharge.
You can use that file to tell the next terminal to unload whatever the file mentions. Then load whatever customers handed into the terminal and have the terminal handle the new stowage layout.
Thanks to loads of practical problems terminals are pretty capable to handle loads of things for you.
Obviously if the terminal itself is also affected it's a way bigger problem.
Hopefully they all got rewarded with, at minimum, a few days off to relax and recover after everything got sorted out.
At least in one country they held a proper thank you party plus thank you gift certificates. That party was for everyone, not just IT.
"And this was a wake-up call to become not just good —we actually have a plan to come in a situation where our ability to manage cyber-security becomes a competitive advantage."
Sadly, most C-level folks don't understand that even gold-standard cyber security is far from perfect, and IT can't do much against dumb, careless users.
> kernel 2.6.24.2, which was compiled back in 2008.
Next you'll tell us XP is vulnerable. Like, Linux isn't 100% immune, but it has a head start.