It's far more complicated than that. For example, there's a lot of strict regulation around how many hours crew can work, how much rest they need between, etc. Likewise planes can't just randomly fly where-ever they want, whenever they want. There's mandatory maintenance, etc. You also have a limited number of hangers and jetways at each airport, so you have to coordinate how planes move around to not overload things. Oh we need to be sure there's enough fuel, fresh oxygen bottles, and so many other things.

The problem is nothing like writing a script to scrape ADS-B data. This is the classic fallacy of a programmer thinking the most cartoonish imagination of the technology is the problem while being completely blind to the fundamental difficulty of organizing large groups of humans in some activity.

The other carriers made it through this high volume period. Do they have better programmers?

The other carriers do also have a tipping point of no return. They were more aggressive about canceling flights in careful waves before they got to that point. Some of the observations here are correct, but there's that big missing piece that Southwest went too far in with an overoptimistic schedule. They won't say that because then there's no easy scapegoat.

There is always a trade-off between robustness and efficiency. By leaving less slack the SWA schedule is more efficient, but also more tightly strung. When failures exceeded the slack capacity the whole schedule fell apart.

The problem of recovery is also extensively studied. But it seems like SW did not put enough effort into having fault tolerant restart.

This whole event is a management decision.

Right. I'm saying though, it's not just the initial state of less slack in their system due to hub and spoke versus point to point. It's also that other airlines were more realistic about backing off flights earlier and more aggressively as a reaction to the developing info on the storm.

It paints a picture of an organisation which didn't see the need to accurately assess risk on a real time basis. If I was the COO I would want rolling projections for the week based on good/bad/ugly weather. But it seems SW was too busy cutting costs and building capacity as cheaply as possible to think about that.

No, as the article explains they use more of a hub and spoke dispatch pattern, which has less chained dependencies.

Yes. Globally the vast majority of airlines use either Amadeus, Apollo, or SABRE for everything from reservations to crew scheduling. United wrote Apollo, American wrote SABRE. Southwest pushes ticketing data to those three, but as of a few years ago they were doing things by hand (like tracking bags and likely scheduling) that other airlines had already automated. If anecdotes from crew are accurate the other problem is that Southwest did nothing to optimize their disaster recovery plan (if there even is one). Manual data entry can scale more than it is at Southwest, but they're still stuck in the teensy airline state of mind where even wildly inefficient workflows can scale.

As for ADS-B, that's hopelessly naive. ADS-B tracks planes, not people. The problem Southwest is having is trying to figure out who is legal to work and where they are. Tracking equipment is so trivial in comparison it's largely a secondary concern.

Modeling and simulation has the concept of Emergent Behavior[1].

Once that complexity tipping point is reached (and other comments suppose that SA competitors were more aggressive at cancellations, avoiding that point in their systems) the system takes on a life of its own.

What follows, for those running the system, is called a "Significant Emotional Event".

[1] https://en.wikipedia.org/wiki/Emergence

>then the problem could be simplified to getting every plain and every crew home to where they would spend the night under a normal schedule.

I'm confused, maybe my calibration of the scale involved here is wrong, but how do you not see this process taking days? Even if you had perfect information of where all the planes and crews were and where they should be, just coordinating with various airports to schedule the flights would take days; and that's assuming every plane is (1) flying empty, (2) fueled and (3) isn't carrying any passenger luggage.

As others have said, you are ridiculously oversimplifying the process.

If you are running a little video game that plots flight charts, sure perhaps you could write such a script. You cannot, however, script a bunch of logisitics support together across disparate, independent, multinational airports.

Add the obligitary paperwork (we don't really want to be shipping people around in coffins, enabling human trafficking or illegal substance smuggling, trust me), and add your standard bit of managerial incompetence managed from an excel document and sharepoint, and it's actually downright impressive if it's only a couple days to completely reboot the system.

Arguably, management doing the standard "oh that will never happen" is probably why it's not even better - you would think the airports would be able to automatically "fix" themselves with a self healing protocol, but that was probably deemed too expensive and left out of the feature set.

Of course I’m oversimplifying the process. But my point is the actual number of flights, passenger and luggage loads and unloads, etc can be a lot lower than a normal workload. On a normal day, an average Southwest plane flies quite a few hops, loads many passengers, unloads many passengers, etc. For a worst case reset, the normal routing doesn’t need to be followed.

Of course this will introduce complexity. But it will also need decently less than a normal day’s number of operations.

If I were designing a solver for where to move aircraft and crews (which is apparently what SkySolver does), I would test it on random and malicious inputs. If it’s initialized with every crew member and every plane at an independent random airport in the US (where Southwest has service) with a few broken for good measure and a few airports shut down, it should still find a solution. Waiting 12-24 hours to start so none of the crew is timed out could be part of a valid solution.

It’s still probably not that simple; the problem seems like it can be analogous to some sort of knapsack and/or traveling salesman problem, which are NP-hard! Even testing would take a long time.

That’s true if you want an optimal solution. If you just want a decent solution (for various definitions of decent), the complexity may not be too bad.

I'm downvoting.

It's simple to say, "oh it would be this easy..." and not consider a single real life scenario.

> problem could be simplified to getting every plain and every crew home to where they would spend the night under a normal schedule

This alone would take days under your own recommendation.

Weeks

"If I ignore all real-world constraints and all the regulatory burdens, it's TRIVIAL. Why can't they figure this out when I could solve it in an internet comment?"

He's ignorant and naive and disrespectful of the workers involved.

Well, let's see... A bit of internet searching shows SWA has about 60,000 total employees. Roughly 6,000 of those are pilots. So probably about 10,000 cabin crew (3 or 4 flight attendants, 2 pilots per flight). Most of those 15,000+ people need to be in exactly the expected place at the expected time, or things will get snarled up. If any one of those 5 or 6 people is a no-show then the plane can't legally fly passengers. There's limited fungibility; one of the pilots has to be a Captain. You don't want two Captains flying, if for no other reason than some other flight will then be short a Captain. One of the cabin crew has to be a Purser (supervisor). It's really preferable to have at least one of the pilots flying to the airports they usually frequent. Off-shift crew still need to be in the proper place at the proper time, or the plane has to stop mid-cycle. Then there's the ground and gate crews.

And you can't just say "go to your usual starting airport". Flights are 24/7. Days of the week matter. Holidays matter. Even your local burger joint doesn't have cast-in-stone schedules. Much less a huge interconnected network that's trying to reboot. Even if 80% of the crew could go to a "usual airport", how would you know if you were in the 20%? Oh... everybody has to phone in, or the system has to send a huge number of notifications out. System crash. (Remember, normally the system depends on scheduling in advance, and only a few last-minute changes need to be handled.)

And how do these what, 20k+ people get where they should be? If they aren't already at that airport, then normally by hopping on a SWA flight. Which aren't happening. So now what? Book flights on other airlines? Who's going to be doing the booking? How does it get paid? How many employees have a company credit card (any?) Use the employee's cards? How many are maxed out for Christmas?

So let's suppose you're scheduled everyone and everybody knows where there's supposed to be and when. What happens when 10% call back in (without crashing the system!) and say they can't get there on time. (Remember, many SWA customers are stuck and are having a hard time arraning alternate travel. Crew without access to SWA flights would have the same trouble.) You're going to either apply massive changes to the existing schedule, or start all over and reschedule everyone. Which is exactly the problem they're currently having. They can't handle massive corrections. What's worse, a crew member may think they're good, and then their travel gets delayed somehow.

How sensitive is the scheduling? If 10% of the crew are no-show or delayed then about 9% of the planes are affected; every 8 hours 90 planes will have the following 8 hours of flights cancelled or delayed (throwing further monkey wrenches in the schedule). About 40 of those planes will be missing a pilot, which means they can't even be deadheaded to where they're supposed to be next.

So how to reboot? They've had to cancel two-thirds of their flights, so apparently they're able to keep 1/3 of them going. Keep those flying so they can shuttle crew around. You're initially only scheduling 1/3 of the crew, so the poor overloaded system can handle it. 2/3 of the flights are just outright cancelled, days in advance, so the customer support load is reduced. ("I'm sorry, your flight has been cancelled. We can't reschedule you until x days from now." vs. much back-and-forth trying to find something with an overloaded system.) Slowly add additional crew and flights so the number of phone-ins is kept manageable.

As you said, this is certainly a complicated situation.

But this is HN, and, from an optimization / constraint satisfaction problem perspective / scale perspective, it’s really not a very large problem. The “going home” problem is a standard programming contest problem. This is more constrained, but a globally optimal solution isn’t needed. Southwest literally has a program called SkySolver. It should work.

So you need to choose destinations for a few thousand planes? Pick the place you would normally have them at 4 AM two days hence. Need captains familiar with the airports? Surely the captains who would normally fly from a given airport at 4 AM two days hence are familiar with those airports. This gives a good starting solution for further optimization.

You need to get a message to ~20k people? Great, Twilio will do that with minimal effort.

Will a budget of a couple million dollars, this is not hard. There are good SMT solvers available for free. CPLEX is expensive but not on this scale. Managing test cases on this scale is straightforward.

Even in an emergency, something good enough to get to the point where SkySolver starts working again should be doable.

People who manage electric grids have protocols for “black starts”. These protocols are messy and complicated, but they are developed in advance, and they work. Airlines should have the equivalent capability.

You touched upon the core issues. Yes, SkySolver should be able to get a solution quickly. Yes, there should be systems in place to co-ordinate invidividual actions of 20k people with rapid adjustments. Yes, they should have procedures in place to do "black starts". But they don't. And you can't design, test, and roll out even a new communications system to 20k users literally overnight. If they existing scheduling system doesn't already email crew, then do they have email addresses for all them? If so, it's probably in HR somewhere; if that's outsourced then it could take days for SWA to get that pulled out even on an expedited basis. If they send email, what percentage of crew will receive it? In what timeframe? Email campaigns are notoriously porous. And how do crewmembers provide feedback? A standalone web page? Great--how do you authenticate? By interfacing the existing system? What will that take? Probably not, though--it appears the existing feedback mechanism is a manual phone call to a person. Do crewmembers even have login IDs?

Finally, even with perfect communication there's the problem of getting people where they need to be when the transport system is very flaky. I think that's probably the issue when the CEO said 'they would get things manually set up, and then something would happen and we'd have to start all over again.'

The pilot's union and others have been critizing SWA for not investing in infrastructure. The previous CEO, responding to the previous melt-down, said something about 'you can't test for these kind of scenarios.' That's wrong. You can't do it easily or cheaply, but it is possible. They haven't done it, they haven't developed robust systems, and now they can't instantly resolve the problem.

> But this is HN, and, from an optimization / constraint satisfaction problem perspective / scale perspective, it’s really not a very large problem.

i can't even begin to imagine what the utility function _alone_ must look like.

the solver can't just produce a blank slate solution every time, it's got to optimize... something, subject to some constraints about not changing too much from the previous plan it came up with. and it's probably got to do this every time there's new real world input.

> The “going home” problem is a standard programming contest problem.

the hubris and handwaving on here is hitting epic levels.

  this is certainly a complicated situation.

uhh

  it’s really not a very large problem

Ok.

  So you need to choose destinations for a few thousand planes? 

No. You need to figure out which planes are where, which of those planes are legal to fly out of which airports, which planes can be made legal to fly with low effort, and which planes need to go in for service (potentially requiring a non-revenue flight).

Ostensibly Southwest operates a fleet of Boeing 737s with two subfleets (ETOPS and non). The ETOPS planes can be used anywhere but the others cannot be used to Hawaii. Some of the ETOPS routes require the range of the MAX and some do not. Some of the airports Southwest flies into can only handle the smallest planes. It's entirely possible that everything stuck in Oakland is too big to fly into Burbank for instance. Or its entirely possible that all of the ETOPS fleet is stuck in Hawaii (so all of the routes too Hawaii are no-gos).

Then you've got to figure out what's broken on each plane. Depending on what's inoperative a plane may not be able to fly into a specific airport if the weather is anything but perfect. If enough stuff is broken the plane may be stuck away from a maintenance base and illegal to fly in revenue service. There goes a plane and a flight crew. These are laws, not optimizations.

And finally you need to figure out where all the diverted flights went. No idea what's normal but Southwest had to divert two flights today (one due to mechanical issues and one due to an unruly passenger). So there's at least one plane that's out of position and unable to fly until it's been repaired.

  Need captains familiar with the airports? 

No. In fact I'm pretty sure that unlike RyanAir and EasyJet, Southwest doesn't fly into any airports that categorically require specific familiarity and training beyond the ETOPS certs required for the Hawaii routes. You do need to figure out who's legal to fly in revenue service and who might be legal to fly on a ferry permit though. Southwest was pretty late to the autopilot game but at other airlines some captains are qualified to land in worse weather than others. So even if you have a flight crew ready to go they may be unable to complete your desired route today – and that's been a problem. The Southwest scheduling software assumes that every pilot completes their assigned flights successfully. Whoops.

Then you've gotta do the same with the cabin crew. And then you've gotta make sure none of this runs afoul of the various CBAs.

And you haven't even addressed the issue of out of position luggage. It's not just that luggage is piling up at airports, but Southwest's actually flown some of the luggage without the passengers.

When that's all said and done you're going to need to handle everything that goes tits up as service resumes. As I pointed out earlier, Southwest had at least two diversions today. So now you have two more out of position planes, crews, and more out of position luggage and passengers.

It's a massively complex, dynamic problem. If a couple million bucks would solve things, Southwest would've spent it already. List price on a single 737 is around $40 million. Even if Southwest got half off you're still talking tens of millions of dollars per plane.

  People who manage electric grids have protocols for “black starts”. These protocols are
  messy and complicated, but they are developed in advance, and they work. Airlines should
  have the equivalent capability.

Most airlines do but, based on the anecdotes from Southwest crew, Southwest does not. Again it's not really that Southwest flies point-to-point, and it's not just a software issue at this point. Procedures that worked when Southwest was a scrappy little airline simply don't scale and Southwest management is ill-equipped to handle this. It's all compounded by the employee contempt that Kelly's managed to foment.

I think the guy you are replying to does not realize that all these are not data in some MySQL database to "use" to solve this quickly.

To be fair it seems like Southwest management fell for the same trap.

I mean, let’s assume that this problem is as simple and everything else on the ground and with scheduling works perfectly (which it isn’t and everyone here is rightly calling you out on. Ever work in an operations environment? Shit goes wrong fast).

You still have to contend with the laws of physics. AKA the speed of aircraft and distance between airports. Compound that thousands of planes making potentially cross country trips and it will take days to do what you are suggesting, leading to the same problem SWA is already facing.

Something worth pointing out here is that according to levels.fyi, a tech lead there who has worked for 20 years has a TC of 174K. It’s unlikely their technical talent would be up to what you propose which is related to the apparent decision at Southwest to underinvest - by their own reckoning - in technology.

This again the HN tech bubble. That TC is around average for most senior software engineers in most corporations in most US cities in the US.

And getting into a tech company and having high compensation doesn’t require “top talent”. Just the ability to memorize DS&A (junior to mid) and memorize system design and being able to regurgitate answers to behavioral questions showing “scope” and “impact”.

It takes knowing two or three well known algorithms to implement the scheduling problem they are trying to solve and system design chops to know how to scale it.

But, as far as having to be on hold for hours, that’s a staffing issue.

Most of their issues are symptoms of poor management - not software engineers.

To be clear, I don’t think you can blame these software engineers. Management has decided the average value of a tech lead to them is about 175K and they are getting that quality of engineer. I think their current tech issues suggest that the value management is placing on engineers is lower than it should be.

Like I said earlier in the thread - there are many bad engineers that are overpaid and many good engineers that are underpaid. On average though, compensation and quality are correlated. In the absence of additional information, compensation is the best metric we have to asses the quality of the underlying engineering talent. It also passes the sniff test - try using the Southwest Airlines app!

Again, you are just as deep in your bubble as everyone else. There are 2.7 million developers in the US. The vast majority of them are paid between $80K-$170k. They are working in banks, government, smaller startups, etc.

And it doesn’t take a $400K senior developer that can reverse a binary tree on the whiteboard while juggling bowling balls while riding a unicycle on a tightrope to make a mobile app.

How much do you think Delta developers make in Atlanta (hint: they aren’t making over 200K). Their app is excellent.

It never ceases to amaze me how little most people on HN know about compensation throughout the industry.

1) These types of comments always make me cringe because it's built on an assumption that everyone is solely trying to optimize for income. I know plenty of people who would take a pay cut to continuously work on what they consider more interesting problems than, say, a social media app or basic CRUD app.

2) COLA matters. $174k in, say, Southwest HQ in Dallas is roughly equated to $422k in SV.

It’s ridiculous to suggest that compensation isn’t correlated with quality. There are many exceptions but it’s a good rule of thumb in the absence of additional information. It also just passes the sniff test - try comparing the Southwest Airlines app to the united airlines app.

I don’t think COLA matters all that much. The reason SV pays such high salaries reflects the underlying higher productivity of those engineers. The fact that much of that income goes to landlords in the Bay Area is orthogonal to the underlying quality of the engineers.

Nobody is claiming they aren't correlated. The question is: how strong is the correlation? The OP seems to insinuate they are extremely correlated on account that it is the sole consideration in the comment.

Like so many strong claims made online, it's an overly simplified mental model for something that is much more complex in reality. Like you said, we need more information before making strong claims.