I figure most companies are too small for that to be budgeted. Though, it’s possibly a good selling point for cloud if it’s capable of it.

Cloud doesn't solve badly designed processes or poorly written software, which seem to be at play with Southwest. Yes, it can help provide more stable infrastructure and there are some (but by no means all) black swan events that can be mitigated simply by throwing more kit at the problem during a surge but it's no silver bullet.

>Seems like this practice needs to expand to more industries.

I think you've mistaken this for something immediately increases quarterly gains with no regard to long-term strategy.

What's most annoying is that there's plenty of employees on the front line who not only care about testing for this sort of thing, but it actually interests them, they're motivated by it, and they understand the dire reality of what happens – to them, primarily – if the company isn't prepared to handle it.

And you can guess what their managers' response typically is: "We need to focus on OKRs and QBRs and KPIs right now... maybe next quarter"

I'm fully convinced that achieving 'manager status' is directly correlated to cowardice. Companies need top-down decision-making, but those decision-makers need to spend more time on the front line.

> Companies need top-down decision-making, but those decision-makers need to spend more time on the front line.

This is not rewarded so it doesn't happen. Managers are rewarded for line goes up so they only focus on line goes up. If line ever doesn't go up it costs them money (advancement, compensation) even if there's little they could have done to make line go up.

In that case the government should smash Southwest with a billion dollar fine so the cost of not doing this drilling exceeds the cost of doing it.

Asking the government to step in for additional regulation is rarely helpful. For this type of failure, the free market will determine whether processes and tools improve, or whether the status quo is good enough.

Free market?

Regulation sometimes helps remind the free market that fuckups like this can come with real human costs.

Regulation imposes those costs and more on real humans and rarely heads off fuckups.

Market forces can correct here. Vote with your wallet.

What's an airline got to do with the free market? They're a extremely highly regulated business.

Until regulation steps in.

My wife and I spent a few hours this morning dealing with the cancellation of our return flight. Southwest has long been preferable for me in many cases, including my most flown route. Between the headache of this outage and the apparently dismal state of their operations there's a strong chance I never fly with them again.

Our family was in San Jose last week when our Southwest flight was cancelled.

It used to be that my first priority would be to go into the terminal and try to talk to somebody. I figured they were the experts. From what I've read, the staff use an antiquated system that takes you from one airport to another, then they can try to get you from that city to where you want to go. That's why there's so much tapping of keys and why it takes so long.

It's better to present them with a route that you've found on Google Flights or similar. The Southwest first flight out was supposed to be yesterday evening, the day after Christmas. In our case, the only thing we could find before Christmas was getting us from SJC to Seattle via Phoenix on Alaska. We ended up renting a car and driving home to Portland. Things got bad around Eugene - I stopped counting after 40 wrecked cars and semis - and got worse as you got closer to Portland.

Airlines don't have quarterly gains. They regularly go bankrupt and get picked up again, because the country needs airlines and because they have large union contracts.

See any gains here?

https://www.google.com/finance/quote/LUV:NYSE?window=5Y

Training for these scenarios may help with responding to true black swan events, like Rick Rescorla's WTC evacuation drills ahead of 9/11. But, nitpicking, if you've predicted something will happen, by way of simulating it, it's not a black swan when it does.

There are cost considerations. Business continuity costs money. Finance firms have significant capital and income to have empty but built out building around around airports for business continuity. Which doesn’t even make sense since they can work from home as proven with covid. Airlines can’t work from home.

Google is also a trillion dollar company, as other have pointed out Soutwest is a low-cost carrier which most probably doesn't have the luxury of hiring FAANG-level engineers on 500k yearly comp in order to best simulate "black swan" events.

Designing and validating the systems that are critical to your daily operations is not a luxury.

Seasonal weather disruptions are not a black swan.

Southwest is not a tiny business, it is the 2nd largest US airline. This is not their first rodeo.

This is probably the best argument for AWS/GCP/Azure even though it is becoming more and more obvious you don't really save that much money.

If you have a black swan event like this and you listened to your solutions architect you will have a disaster recovery plan or even better a multi region setup. Worst case you have highly paid support engineers at the cloud providers who will do everything they can to get you back online.

This does not seem like a hardware failure scenario where the cloud has anything to offer. More like their intricate software/database systems became out of sync with reality and disentangling the mess is a highly manual process.

In this case no, but I was more referring to the 2016 delta ground stop that was due to their datacenter burning to the ground.

I remember reading about the Delta incident a ways back, here they claim it cost them ~$150 million. https://www.datacenterknowledge.com/archives/2016/09/08/delt...

That's not the article I hoped to find however. I seem to remember there was another article where they hired a investigator/consultant to figure out the price to migrate to the cloud and ensure "this never happens again."

My recollection of that was: their scheduling/ops team is also in the same city (Atlanta GA) as this datacenter, and that teams work was brought to a halt by the datacenter outage. The investigator concluded that Delta would need redundant copies of the ops team or the whole effort of moving the software to the cloud would just be at risk to something happening to the human team all in the same city. That would obviously cost to much money, so Delta decided to skip it.

They don't seem to pay competitively with banks, let alone FAANGs, though the benefits and culture are (or were) reputedly fantastic.

Source: Am a local who's been headhunted by them a few times but never got beyond the initial discussion with the headhunter for this reason.

Regarding the employees, keep in mind that neither SWA (nor any other airline for that matter) have big software engineering departments. It's all outsourced to either generalist bodyshops for custom/peripheral systems (IBM, Accenture) or specialist shops for core (Amadeus, SABRE)

20 years ago I used to work for an airline. Back then Sabre was nothing more than the mainframe in Tulsa. All the APIs did nothing more than perform automated green screen commands. Has anything changed with Amadeus or Sabre? Or is it still mainframes behind the curtain?

There are two parts to the IT: the airline backend and the distribution backend. For the airline backends, Sabre used to build custom single-tenant ones per airline, on mainframes. The SWA one was very old, hence their inability to charge for bags. The Sabre distribution system is common of course, again on mainframes

Amadeus are eating them up, because their airline backend system is a shared multi-tenant setup, built on commodity hardware. Their distribution system used to be mainframes, but they managed to migrate away in the 2010s.

Sabre is still alive, but only in North America, and Amadeus is slowly chipping away (WN, AC..)

Thank you

Yandex used to run datacenter loss training every week, where they will nullroute one DC and see what breaks all while handling live taffic.

At Facebook we would simulate an entire datacenter disappearing.

When we first started doing it the datacenter would be chosen months in advance so that teams would have plenty of time to ensure their services can run without that specific datacenter.

When I left this year, the datacenter would be randomly chosen on the same day it would be cut off.

That's pretty cool and ideal practice for a software firm but in one of the reddit threads they're talking about mass quits/refusal to work of ground crew at Denver because of the weather. I wonder how you could ever prepare for that? Keep a backup, airport scoped, ground crew in the waiting room??

You can't really do hot spares for people without time to gear/train up and the weather event is so widespread I doubt there's enough spare SWA human capacity across the whole nation even if you had C130s on standby everywhere ready to take workers where they're needed most. From a national security perspective, situations like this is why the Marines exist right? Ensure a rapid response while the rest of the machine gets moving. I feel bad for everyone involved, those affected and those trying to figure out a solution.

> I wonder how you could ever prepare for that?

Management could consider how pay and performance programs can help ensure business continuity.

HR and MBA xls wizards don't understand how to manage for business longevity.

Pay might not really be enough. Maybe management could try to find folks to babysit kids/take care of parents trapped at home, freeing up workers to come fly. They'll certainly fail, but at least they'll understand the plight of their workers.

Strangers paid on the lowest cost contract to leave with your kids?

What's the big deal? You can always make more.

/s

Do you really think the MBA wizards can't figure out some basic pay issues?

It seems you're discounting just how complex HR can be, especially in the face of exigent circumstances. No amount of bonuses will immediately staff up an entire terminal in the face of a massive snowstorm.

While they may be able to figure it out, optimizing pay to quality of life at work ratios to ensure long term employee retention and loyalty has certainly not been a priority.

> No amount of bonuses will immediately staff up an entire terminal

That is right and thus I would engage line management to figure out business continuity.

> in the face of a massive snowstorm.

It is winter. The storm was tough but not exceptional for the season of winter. Denver did not report tremendous amounts of snow.

Few if any MBAs can get on the ramp and look a line employee in the eye and lend a hand. The wfh keyboard warriors don't know blue collar and therefore are unable to figure this out. MBAs can figure out ways to game their pay. Shareholders should demand some senior management unemployment applications.

> The storm was tough but not exceptional for the season of winter.

Sure seems like this was an exceptional storm. Widespread, deep cold reaching into Mexico, snow falling across much of the U.S., Buffalo hit with the most snow in 20 years, records set multiple locations.

A couple decades of weather variation is not exceptional, though some will disagree with me when their bonus is yearly. I recall it was cold in Texas last winter. Weather is seasonal and varies.

I wonder if board of directors will compare performance with other airlines.

It doesn't seem that farfetched for an airline to run a drill where a given airport is assumed inoperable to see how the system reacts. The expectation shouldn't be the same as the data center failure but you can learn what you aren't doing well enough.

You can certainly handle it better than SWA is handling it. Maybe if they’d actually run a simulation where flights out of one or more cities had to be wholesale canceled, they would be. You can’t fix the situation for everyone, but you can avoid fucking up this badly.

Google is made of money, and the reason they are is not because of DiRT. Other industries can't afford the same things that Google can while continuing to be a business.

After canceling 5,400 flights, I can't see how Southwest can afford not to test. Even if they only made $1000 off each flight, that's still $5 million they just lost.

They probably would not have made it this far if they tested for every possible scenario, their margins are razor thin.

Anyway, doesn't even the best testing only catch 40% of bugs or thereabouts? It's not a silver bullet.

It's not any bullet, but spending a week each year to do downtime testing is going to expose more issues than not spending any time on it at all.

This does not need Google deep pockets. It needs the motivation and some funding. SWA does not care.

Pull the backup tapes, hand those to DR team, provide bare metal, and start the stopwatch. I participated in this in 1990s across the Mississippi.

Personally one of the basic tenets of my adulthood is realizing how many companies are a hair away from a similar scenario (differing in magnitude from an airline ofc).

EMP FTW

That... and Chaos Engineering in general, and also just generally a forward looking team that identifies threats, vulnerabilities, and risks in the future and works backwards to identify potential mitigating controls.

(Look forward, reason backward).