News for hackers. Stuff that matters.
Re:Southwest cancels 5,400 flights in less than 48 hours by paganel on Tuesday December 27, 2022 @01:16PM (#34151729)
Google is also a trillion dollar company, as other have pointed out Soutwest is a low-cost carrier which most probably doesn't have the luxury of hiring FAANG-level engineers on 500k yearly comp in order to best simulate "black swan" events.
Re:Southwest cancels 5,400 flights in less than 48 hours by blevin on Tuesday December 27, 2022 @01:56PM (#34152293)
Designing and validating the systems that are critical to your daily operations is not a luxury.

Seasonal weather disruptions are not a black swan.

Southwest is not a tiny business, it is the 2nd largest US airline. This is not their first rodeo.

Re:Southwest cancels 5,400 flights in less than 48 hours by makestuff on Tuesday December 27, 2022 @02:00PM (#34152346)
This is probably the best argument for AWS/GCP/Azure even though it is becoming more and more obvious you don't really save that much money.

If you have a black swan event like this and you listened to your solutions architect you will have a disaster recovery plan or even better a multi region setup. Worst case you have highly paid support engineers at the cloud providers who will do everything they can to get you back online.

Re:Southwest cancels 5,400 flights in less than 48 hours by 0cf8612b2e1e on Tuesday December 27, 2022 @02:15PM (#34152537)
This does not seem like a hardware failure scenario where the cloud has anything to offer. More like their intricate software/database systems became out of sync with reality and disentangling the mess is a highly manual process.
Re:Southwest cancels 5,400 flights in less than 48 hours by makestuff on Tuesday December 27, 2022 @02:17PM (#34152563)
In this case no, but I was more referring to the 2016 delta ground stop that was due to their datacenter burning to the ground.
Re:Southwest cancels 5,400 flights in less than 48 hours by therealjumbo on Tuesday December 27, 2022 @04:49PM (#34154323)
I remember reading about the Delta incident a ways back, here they claim it cost them ~$150 million. https://www.datacenterknowledge.com/archives/2016/09/08/delt...

That's not the article I hoped to find however. I seem to remember there was another article where they hired a investigator/consultant to figure out the price to migrate to the cloud and ensure "this never happens again."

My recollection of that was: their scheduling/ops team is also in the same city (Atlanta GA) as this datacenter, and that teams work was brought to a halt by the datacenter outage. The investigator concluded that Delta would need redundant copies of the ops team or the whole effort of moving the software to the cloud would just be at risk to something happening to the human team all in the same city. That would obviously cost to much money, so Delta decided to skip it.

Re:Southwest cancels 5,400 flights in less than 48 hours by 0xBDB on Tuesday December 27, 2022 @03:04PM (#34153099)
They don't seem to pay competitively with banks, let alone FAANGs, though the benefits and culture are (or were) reputedly fantastic.

Source: Am a local who's been headhunted by them a few times but never got beyond the initial discussion with the headhunter for this reason.

Re:Southwest cancels 5,400 flights in less than 48 hours by namdnay on Tuesday December 27, 2022 @04:31PM (#34154138)
Regarding the employees, keep in mind that neither SWA (nor any other airline for that matter) have big software engineering departments. It's all outsourced to either generalist bodyshops for custom/peripheral systems (IBM, Accenture) or specialist shops for core (Amadeus, SABRE)
Re:Southwest cancels 5,400 flights in less than 48 hours by rawgabbit on Tuesday December 27, 2022 @06:36PM (#34155423)
20 years ago I used to work for an airline. Back then Sabre was nothing more than the mainframe in Tulsa. All the APIs did nothing more than perform automated green screen commands. Has anything changed with Amadeus or Sabre? Or is it still mainframes behind the curtain?
Re:Southwest cancels 5,400 flights in less than 48 hours by namdnay on Wednesday December 28, 2022 @01:33AM (#34158124)
There are two parts to the IT: the airline backend and the distribution backend. For the airline backends, Sabre used to build custom single-tenant ones per airline, on mainframes. The SWA one was very old, hence their inability to charge for bags. The Sabre distribution system is common of course, again on mainframes

Amadeus are eating them up, because their airline backend system is a shared multi-tenant setup, built on commodity hardware. Their distribution system used to be mainframes, but they managed to migrate away in the 2010s.

Sabre is still alive, but only in North America, and Amadeus is slowly chipping away (WN, AC..)

Re:Southwest cancels 5,400 flights in less than 48 hours by rawgabbit on Wednesday December 28, 2022 @10:20AM (#34161810)
Thank you