https://www.dallasnews.com/business/local-companies/2016/08/...
at the time, then-CEO now-chairman Gary Kelly said:
> "What's unique is the partial failure, it's never happened," he said. "This isn't a drill you can run."
https://web.archive.org/web/20161112192103/http://www.dallas...
Delta had a similar outage due to a datacenter fire, grounding all domestic flights. Southwest was uniquely slow in taking days to start up again. And if the way my American Airlines ticket switched my birthdate to January 1st, 2000 is any indication, many airlines still need to modernize.
I think you've mistaken this for something immediately increases quarterly gains with no regard to long-term strategy.
And you can guess what their managers' response typically is: "We need to focus on OKRs and QBRs and KPIs right now... maybe next quarter"
I'm fully convinced that achieving 'manager status' is directly correlated to cowardice. Companies need top-down decision-making, but those decision-makers need to spend more time on the front line.
This is not rewarded so it doesn't happen. Managers are rewarded for line goes up so they only focus on line goes up. If line ever doesn't go up it costs them money (advancement, compensation) even if there's little they could have done to make line go up.
Market forces can correct here. Vote with your wallet.
It used to be that my first priority would be to go into the terminal and try to talk to somebody. I figured they were the experts. From what I've read, the staff use an antiquated system that takes you from one airport to another, then they can try to get you from that city to where you want to go. That's why there's so much tapping of keys and why it takes so long.
It's better to present them with a route that you've found on Google Flights or similar. The Southwest first flight out was supposed to be yesterday evening, the day after Christmas. In our case, the only thing we could find before Christmas was getting us from SJC to Seattle via Phoenix on Alaska. We ended up renting a car and driving home to Portland. Things got bad around Eugene - I stopped counting after 40 wrecked cars and semis - and got worse as you got closer to Portland.
See any gains here?
Seasonal weather disruptions are not a black swan.
Southwest is not a tiny business, it is the 2nd largest US airline. This is not their first rodeo.
If you have a black swan event like this and you listened to your solutions architect you will have a disaster recovery plan or even better a multi region setup. Worst case you have highly paid support engineers at the cloud providers who will do everything they can to get you back online.
That's not the article I hoped to find however. I seem to remember there was another article where they hired a investigator/consultant to figure out the price to migrate to the cloud and ensure "this never happens again."
My recollection of that was: their scheduling/ops team is also in the same city (Atlanta GA) as this datacenter, and that teams work was brought to a halt by the datacenter outage. The investigator concluded that Delta would need redundant copies of the ops team or the whole effort of moving the software to the cloud would just be at risk to something happening to the human team all in the same city. That would obviously cost to much money, so Delta decided to skip it.
Source: Am a local who's been headhunted by them a few times but never got beyond the initial discussion with the headhunter for this reason.
Amadeus are eating them up, because their airline backend system is a shared multi-tenant setup, built on commodity hardware. Their distribution system used to be mainframes, but they managed to migrate away in the 2010s.
Sabre is still alive, but only in North America, and Amadeus is slowly chipping away (WN, AC..)
When we first started doing it the datacenter would be chosen months in advance so that teams would have plenty of time to ensure their services can run without that specific datacenter.
When I left this year, the datacenter would be randomly chosen on the same day it would be cut off.
You can't really do hot spares for people without time to gear/train up and the weather event is so widespread I doubt there's enough spare SWA human capacity across the whole nation even if you had C130s on standby everywhere ready to take workers where they're needed most. From a national security perspective, situations like this is why the Marines exist right? Ensure a rapid response while the rest of the machine gets moving. I feel bad for everyone involved, those affected and those trying to figure out a solution.
Management could consider how pay and performance programs can help ensure business continuity.
HR and MBA xls wizards don't understand how to manage for business longevity.
/s
It seems you're discounting just how complex HR can be, especially in the face of exigent circumstances. No amount of bonuses will immediately staff up an entire terminal in the face of a massive snowstorm.
That is right and thus I would engage line management to figure out business continuity.
> in the face of a massive snowstorm.
It is winter. The storm was tough but not exceptional for the season of winter. Denver did not report tremendous amounts of snow.
Few if any MBAs can get on the ramp and look a line employee in the eye and lend a hand. The wfh keyboard warriors don't know blue collar and therefore are unable to figure this out. MBAs can figure out ways to game their pay. Shareholders should demand some senior management unemployment applications.
Sure seems like this was an exceptional storm. Widespread, deep cold reaching into Mexico, snow falling across much of the U.S., Buffalo hit with the most snow in 20 years, records set multiple locations.
I wonder if board of directors will compare performance with other airlines.
Anyway, doesn't even the best testing only catch 40% of bugs or thereabouts? It's not a silver bullet.
Pull the backup tapes, hand those to DR team, provide bare metal, and start the stopwatch. I participated in this in 1990s across the Mississippi.
(Look forward, reason backward).
The thought was that it needs to be something that isn’t believable to a non-technical user seeing it on their computer screen. It turns out that this is not necessarily useful. I listened to a guy talking about some issues with a record; he says “1871? What’s up with that?” And then just moved on as if “well it came out of the computer, must be right” or something.
I think that databases need to have the concept of NaN for dates and time stamps, except that this should be configurable to something like a poop emoji or something like ⁉🆘. It has to be something where your grandma would look at it and confidently say “your computer is broken”
How about just NULL?
Now GLHF!
In some situations, you are trading one known point of failure for a million unknown ones. Among other problems :)
Default date types, with some columns being nullable, work just fine for my project. I wouldn't want them to be more complicated and force me to consider additional cases, especially if those cases aren't language compatible.
Off topic, but this reminded me of:
Hello, I'm Mr. Null. My Name Makes Me Invisible to Computers
As someone who writes some very thorough unit tests... and also have had to have mandatory training... I find "this isn't a drill you can run" to be _very_ wrong.
It would not surprise me that their back-office operations are likewise economized and some things are just not done because "they can never happen."
Southwest isn't a particularly budget airline compared to modern budget carriers like spirit and ryanair that haven't copied the open boarding policy. I suspect the opportunity to upsell seats / luggage and have distinct classes outweighs the turnaround time costs of assigned seating.
Meh, it doesn't even have to be "never". It just has to be cost multiplied by frequency is less than the cost to prepare.
If they lose $100m every five years due to a system failure, and it would cost $30m/year to plan for those failures, they it's just cheaper to let it happen.
And I don't mean this in a judgmental, Fight Club-car recall speech kind of way. It's just business reality. At some point every business has to decide that the cost of planning for something is higher than the cost of letting it happen.
Sometimes businesses end up on the wrong side of that bet. They see only the costs but not the benefits of preparedness (by the time it fails, there will probably be a different CEO in charge) and make a bad call.
The airline industry feels like one where each year it's a different carrier who has some catastrophic scheduling failure. Today, everyone says they're never flying Southwest again. But if you fly semi-regularly then it won't take very long before you don't have any airlines left to fly on.
For people who weren't affected, I doubt very many are even going to remember this. Personally, I remember that this kind of thing has happened recently with other carriers but I couldn't even tell you who.
And people who were affected can mostly be bought off if you need to. Some vouchers & hotel reimbursement and it's just the cost of doing business.
Plus, the airline industry has proven over and over that people are willing to put up with a lot when you have the cheapest prices.
It's different from an industry that's built on reputation and trust. Like, a password manager, the only real thing you're selling is your reputation. Losing trust is a real existential threat. Security costs need to be in the bucket of either "yes, we will do it" or "it's so expensive that if we do it then we don't have a business anyway, so we'll skip it and pray."
I think reputation impacts in this industry from anything other than crashes don't hold much staying power.
I'm actually quite tempted to buy the SWA dip...
They're also the "friendly airline", they easily have the most personable and friendly staff. I don't know what they do different, but Southwest employees treat me human and all the rest generally treat me like human trash. It's got to be a company culture thing, maybe connected to Southwest not having a first-class section.
Usually I fly with Southwest whenever possible without thinking twice about it, but this outage and the outage last year are forcing me to reconsider. Better to deal with rude people than to have my flight delayed..
Definitely. Normal American customer service is to treat you like human trash, so obviously Southwest has decided to do this differently and so probably does things like only hiring friendly people, training them to be friendly and positive even in difficult situations, and checking on them somehow to make sure they're doing this and not just faking it for the interview and training. Other airlines obviously don't, but it's not just airlines, it's everywhere in American customer-facing business these days. Rudeness is just a normal part of America culture now.
I have a backpack that has a phone holder on the shoulder strap. When you click your phone in it, the camera is visible. I've had many a situation where someone was about to go full New York on me, but noticed the camera staring at them and toned it down. I've never once been recording.
training them to be friendly and positive even in difficult situations
That's what happened with Dr. Dao in any case. Not long after that shit show, I booked a transcon on American. Turns out it was overbooked, and they were desperate to get people off the plane. The gate agents basically asked how much money it would take to get people to volunteer to take another flight. They got their volunteers, everyone went home happy because American empowered their staff to resolve problems.
United is awful. American isn't much better.
With the exception of United I'd say most airlines I've dealt with have been pleasant whether or not I'm flying up front or have status. Yes, even RyanAir. We're in something of a golden age since (with the notable exception of Southwest) American airlines have mostly put their mergers behind them and have mostly shed themselves of CEOs who view employees as adversaries. Take a look at the bad old days of the late 80s through mid 2000s. Smisek. Lorenzo. Parker. Bastards and crooks, all of them. It's difficult to stress just how toxic airline leadership was and how that trickled down for a long time.
A fortune, they only just finished an 8-year migration to Amadeus
As a IT-VP/CIO, the statement of "there's no way to test it" is not acceptable.
Thorough unit and integration tests include failure modes. Mine include things like "what happens if the OS reports that the storage was unmounted during read/write" (because that was a failure often seen in production with some flaky SAN devices) and "what happens if the server stops responding" (because networks are generally unreliable) and "what happens if invalid (random) data is given" because data corruption is a frequent occurrence for similar reasons.
> Perhaps you have a false sense of security about what you’ve really been testing?
Perhaps. On the other hand I've seen a _lot_ of other developers test only the happy path and call it a day then spend days/weeks/months debugging failures.
The unspoken part you have to hear there is "... within the economic model of the airline business".
Business continuity gets exponentially more expensive as you chase the blackest of swans: the sheer volume of plan development and maintenance, developing exercises, table-top vs. walkthrough vs. simulation, assumptions about how many different uncorrelated failures you're prepared for deal with at once etc.
I've no doubt you could run an airline to be as resilient as (say) USAF Air Mobility Command, but no-one could afford the tickets.
They only have one geo-located data-centre?
Most of the travel industry runs on old software that would horrify a lot of people here, especially those who've never worked for a large, 30+ year old company. When I used to interview a lot of people I made it a point to mention some of the more "interesting" aspects so they'd know what they were getting into.
One example: ever tried to book a flight a year in advance? On a lot (almost all?) of systems you can't, because the underlying date format is "DEC27".
Major holiday, destination wedding, event known long in advance (e.g. Grandma's 80s does not come as a surprise).
- Annual conferences or conferences that occur every-other year - Planning family reunions because you need that kind of cat-herding lead time when you have 9 uncles/aunts on just ONE side of the family - Periods where I have some spare cash I'd like to lock in a getaway with before I spend it or something unexpected like the invasion of the Ukraine drives up fuel costs and overall prices... or a global pandemic hits - would be sweet if I could have rebooked some of my trips for for 1-2 years out when the pandemic hit - Travel for future medical stuff; at one point for 2-3 years I was taking my mom to the Cleveland Clinic every 4 months for periodic checks and it would have been super nice to be able to just book that stuff way in advance and have it all taken care of
Etc
etc
etc
I'd bet quite a few people would appreciate that ability
Granted, the rollout across airlines is probably glacial
Source: I used to work there
Complexity — Fundamentally you’re looking at a logistics software, except unlike packages you’re dealing with people who aside from expected destinations have travel lengths and time-in-air calculated down to the minute. Also unlike a package, a surprise multi-day trip, unexpected multi-leg journey, one day delay is not something passengers (and crew members) will accept or be at all ok with. And if any one thing goes wrong there’s going to be cascading failures down the line— so much that it may break your company’s entire operating workflow (e.g. Southwest) entirely, and no software can overcome that kind of organizational gap.
Airlines - There’s not many commercial passenger airlines left in the US, especially that fly nationwide. Good luck trying to convince one of these giant behemoths to move to a non-battle-tested system for core operations, especially when decades-old industry software and practices around that software exist.
Also— Sabre’s software itself is actually reliable! As a corporation it is slow clunky and bureaucratic, but the actual functionality it provides is stable, battle-tested, can handle any travel edge case you can think of, and fast and efficient for those who know how to use it, while also good enough at day to day operations that it doesn’t take too much time to train new agents on how to use it for routine tasks.
I think though Southwest's issues are more on their side.
Yeah building a new GDS today is an exercise in insanity, it's a huge complexity nightmare and switching probably impossible. I always wondered if AI could eventually improve things, but the existing GDSs are unlikely to care much to try. It's basically a (tri)monopoly you can never break.
1) Replacing any software for an airline carries huge risk. They are barely operating ok with the software they have and holding it together with duct tape. Even something you might regard as ancillary, like a baggage handling software system, or flight catering software system, if it goes down, has the potential to disrupt thousands of planes and hundreds of thousands of passengers for days.
It is so significant an issue (to try to change some software, and just one out of many systems that have to talk to each other) that if an airline ever considers doing this, they may actually stop operations for some number of days while they do it rather than risk having operations go wrong. There are some rare examples of airlines doing this to try to change their systems.
2) Related to the above, airline management hates to be embarrassed by something that might work but has the potential to go badly wrong. So they are very conservative when it comes to replacing systems that are working, even if it's painful / much less functional than what they might achieve by a change.
Combine these factors (and many others) and it means that sometimes starting a new airline is simpler than trying to fix an old one...
Now there's a truly total rewrite...
Let's ask the opposite question:
Why do you feel that software necessarily needs to be modernized ?
I'm not sure how well SABRE works but I do know how fast and efficient keystroking through a non-GUI interface can be and I don't know why expert mode interfaces should ever be replaced by unsophisticated mousey-mouse-mouse ones.
At least, I feel confident in that analysis, since this exact same issue happened to Southwest in 2016, before they were using Sabre. Which would point to a chronic organizational failure
https://www.washingtonpost.com/news/dr-gridlock/wp/2016/07/2...
https://www.sabre.com/insights/releases/sabre-forges-10-year...
https://en.wikipedia.org/wiki/Sabre_(travel_reservation_syst...
It's not even an ASCII text app, but an EBCDIC one. Or was after EBCDIC was defined as a standard, after SABRE itself launched.
If you can see how it works, it horrifies me even more as a traveller, as from outside it just doesn't work a lot of the time.
Also if you just look at the video, we all know how bad these systems are, but are not able to do anything (starting anything new in the airline industry has too much cost).
6 years old so hopefully something has improved...
When characterized as something that can't be done instead of something they don't know how to do, you know exactly where they are on the Dunning–Kruger curve.
Investment capitalism is really a garbage system when it comes to building and maintaining basic infrasctructure like transportation, electricity grids, roads and so on. China has demonstrated that convincingly over the past two decades, hasn't it?
The difference between what's true, what some people will buy, and what you can get away with saying is gross, y'all.
Netflix and airlines are so different as to make this comparison laughable. The cost of setup and consequence of problems actually being found (ie Federal Regulations) that are not addressable (it's not like SWA didn't know about some of the eventualities), easily outclasses the need for testing every combination of situations. Kong doesn't run anything that has to do with weather turning jet fuel into sludge or 12x pre-staffing in case of massive computer failures along with assessing the possible legal consequences from each locale. The hubris of pretending that physical services on a national scale, is as deterministic as a complex automated system, is unsurprising from a certain crowd, I guess.
* this memo[1] from the SWA VP thats circulating dated December 21st calling a staffing emergency saying all hell is about to break loose. I'm pretty sure that guy didn't know of the software crash before it happened
* Every single piece of checked in luggage is going to take 30+ days to be found? Weird none of the extensive press has mentioned people not getting their bag. Also its pretty much an airport thing and not an airline thing. Airports are in charge of deplaning your luggage and bringing it to you
* They're running 40% of flights completely manually with 0 scheduling software? That's extremely impressive
[1] https://mobile.twitter.com/ruthschmidt/status/16074609858619...
It appears there was a MASSIVE disconnect between baggage and people, bags are ending up all over the place even though the people never left.
Although the airline app has always been accurate it's cheap insurance if a bag isn't properly scanned or something.
It's been great. My luggage missed a connection in december, when I arrived at the destination I could instantly see that it still was at the other airport. So while the others were waiting for the bags, I could already file a complaint (at the nearly empty complaints queue). Then three hours later I could see bag movement to the tarmac and knew that my bag cought the next flight.
Also, when in a city, it's a lot of fun to say: bring me back to my bag instead of looking up the hotel address.
For anyone with an iphone & checked bags I can really recommend it.
I think this is a great idea though and I didn't even think about "bring me back to my bag" - that's awesome :)
It’s got to be less power than the phone uses to stay in contact with the towers.
i believe this story. the scheduling system is too fragile and can’t automatically pick up disruptions to crew movement. they’re still manually scheduling some crews, but I think their main problem is they don’t have a way of knowing which crews are where and therefore can’t schedule flights. and it’s not every single piece that’s going to take 30 days, but it seems pretty likely that some bags will be lost that long.
But hey it's cheaper if we don't change it right? /s
(and I kinda can agree with this, like 10% because they might end contracting with some bigger IT company that doesn't care about shipping something that works)
Modernizing such a system would take enormous political + real capital. The new system would undoubtedly have many growing pains as formerly resolved problems were not covered in the new system. It’s no surprise nobody would want to touch such a system.
I've worked with airline scheduling software at almost the scale of United (about half the size, for the largest customer). That was using an Oracle database, which was a bit of a pain and a big expense, but worked fine as long as there was competent admins and competent devs. Would Couchbase be disaster-proof even when run by clowns?
Not saying there aren't better and worse choices for databases in any given situation. Just saying that there are lots of them that work perfectly well in competent hands.
If they've resorted to doing all their scheduling manually with pen and paper, any data pushed to the edge (air crews, ground staff etc) will need to be transmitted in a similarly manual fashion.
The gate/ground/air crews are not making autonomous decisions about who is flying what, when and where. That is decided by some ops/crew control structure in the airline.
Unions fight to keep a lot of data from being automated instead of manual because the automated data can be used to build individual performance metrics. For example, some baggage handling unions fought against bag scanners because then you could tell how much work different union members did.
They’re trying to keep from getting optimized to death like an Amazon warehouse worker.
Unfortunately that same automated data is really useful for dealing with situations like this. Once the manually entered data is out of sync with the real world the errors compound and you have cascading failures.
That depends on the airline and airport. It’s extremely common for the airline which operates out of a large part of a terminal to do their own ramp work.
Southwest employs thousands of ramp agents to do their own ground ops and under-wing work at their bases.
Why not? Some things aren't hard to predict, it's like a Ticketmaster employee saying "our site is probably going to go down when the Taylor Swift tour goes live". If you know your systems are deficient then you can predict they'll fail in a crisis, especially if that failure has happened before (as it has for Southwest).
This is obviously intentional, and in many parts of the US totally legal. I don't know specific Colorado laws but there is no federal law against this.
Scheduling is not an issue, and there is no need to waste an ED's time on such things. Some urgent care spots specifically have a "doctors note" service.
It doesn't make the policy any better (I've worked such jobs), but I don't think there is a need to exaggerate how the process works in real life.
It looks like this is not a fun week for anyone who works at Southwest.
1. Hiring more people (more than 3 days in advance).
2. Paying people more so they don't quit.
3. Giving people paid sick time and not threatening their jobs when they take it so that they don't come to work sick and get a bunch of their colleagues sick.
4. Selling fewer tickets and running fewer flights if you don't have enough capacity to support the current workload.
An airport subcontractor came to us having issues with getting enough baggage workers. Issue was they paid minimum wage, were located 30 minutes outside the city, and there was no public transport. There was no way for the workers to get to the job! The narrative was "these worker are so untrustworthy/lazy/full of excuses." So they would pay a 50% to an agency hoping gig-contractors could save them on single-day contracts rather than just putting that money to better wages or transit perks.
Contempt really does describe the management attitudes I ran into.
It's worse than contempt, it's abuse. He is looking for ways to make recovery from illness more difficult than it needs to be, and in turn hope some will come to work when they can't get a "regular" doctor visit in time - during a holiday week.
That memo is specifically about ground operations at DEN, and specifically because of the arctic weather conditions and a high number of sick calls and ramp agents that outright quit. You can earn more flipping burgers in Denver than you can as a SWA gate agent with five years of experience. Failing to pay staff is also a management failure.
The scheduling software crashed due to the number of pilots and flight attendants that were out of position and the number of changes that were made to the schedule. I would imagine that there was an overflow in some situation -- i.e. "the number of changed schedules should never exceed 65535" that worked every year until this one. But this system was already known to be unstable, another Reddit comment said that "there are settings you don't change for fear the entire thing will crash." Which it has before in 2016. Not expecting that history will repeat itself and doing something about it is also a management failure.
> Also its pretty much an airport thing and not an airline thing
Absolutely incorrect. While the airport runs the automated conveyance system that gets the bag from where the rampers drop it to the baggage claim, the people that handle the baggage at every manual step in between are SWA employees. If there aren't enough of them, the bags don't make it on the plane on time.
Notice that the people doing this work for Southwest Airlines are all wearing Southwest uniforms.
Like most complicated failures, this was failure with multiple causes and contributing factors. The core of the problem seems to be that management was rent-seeking without making appropriate structural changes to keep up with system load.
Additionally I was told (by a crew member) Southwest refused to meet wage demands with ramp / fuel crews so they all quit. Apparently they sent them an email basically saying a recession is coming so be thankful you have a job... Meanwhile all other jobs pay more in the area. Anecdote I know, but we did have to wait 2.5 hours for fuel on my flight.
https://www.reddit.com/r/SouthwestAirlines/comments/zw6upo/h...
I thought this too, but apparently there are no federal regulations other than reimbursement for the direct costs.
"Airlines are not required to provide passengers with money or other compensation for costs that fall outside of the cancelled airline ticket and fees tied directly to the airline ticket (such as baggage fees, seat upgrades, etc.) when flights are cancelled." [1]
"Each airline has its own policies about what it will do for customers on bumped or cancelled flights. There are no federal requirements." [2]
However, there are rules around "bumping" passengers on a flight.
"An airline is required to compensate you after involuntarily bumping you from an oversold flight in certain situations. However, there are many situations where you are not entitled to compensation." [3]
[1] https://www.transportation.gov/individuals/aviation-consumer...
[2] https://www.faa.gov/faq/what-are-policies-bumped-or-cancelle...
[3] https://www.transportation.gov/individuals/aviation-consumer...
I think some slack/spare capacity or redundancy somewhere would make a LOT of problems less likely to happen. Running lean only works out when nothing goes wrong. I know I should not be shocked by the extent of human stupidity, but why the "business" types do not get this is bizarre to me.
And paying people more would help too.
Software is core to your business. If you don't invest in it, you're going to pay the price.
This is amazing. Southwest was famous for its operational efficiency and quality. Companies eagerly learned from them. I wonder what has changed.
Saved screenshot: https://i.imgur.com/bchgmm7.jpeg
On behalf of all employees: WE ARE SORRY! I will give it to you straight- this meltdown was caused entirely by Southwest. It was triggered by the storm, but the failure to recover quickly is on Southwest 100%. If you are still hearing “weather” almost a week after the storm, it’s not true. Couple main points: 1. Please be patient with us. We desperately want to do everything we can to get you where you’re going. 2. This shitstorm is because the crew scheduling software went belly up and it almost all has to be unraveled over the phone with crew members calling scheduling. If we had better technology which eliminated the need for phone calls, this would have been fixed by now. 3. If you are able to find alternative transportation to your final destination- DO IT. Another airline, bus, train, Lyft, rental car, ANYTHING. Southwest WILL NOT be able to get you to your destination anytime in the next few days. 4. Like I said, it’s gonna take at least a week to get back to normal operations for Southwest. If anyone has questions, I will try to answer them. I work ground ops at one of SWA’s hubs. EDIT FOR FAQs——
Checked bags are currently a disaster. Plan to not see your checked luggage for at least a month. In the interest of 100% transparency, some bags will be 30+ days lost in the system.
Will my flight for X date go out? Next 3 days- plan on a cancellation. 4-7 days- likely to go as scheduled. 7+ days- should see operational recovery.
https://www.reddit.com/r/bestof/comments/zwd1fq/u4sammich_ex...
https://www.reddit.com/r/flying/comments/zw5lsl/southwest_pi...
This has to be really hard as most systems were probably built assuming something that smelled like full staffing. Pilots can only fly X hours, and flight attendants usually can't be forced to take overtime ... there have to be similar constraints on ground crew, maintenance and gate workers, too. So when you are planning for an understaffed airline, it has to be extremely difficult to optimize where the failures will occur.