News for hackers. Stuff that matters.
Re:How Southwest Airlines melted down by TheCondor on Wednesday December 28, 2022 @04:27PM (#34166211)
Is resumption difficult or is it resumption and then make whole the tens of thousands of customers that were supposed to be moved a week prior?

No idea what SkySolver actually does in totality, I'm sure it's complicated but I would think a flight crew could indicate where they are right now and then it could maybe pickup the next possible course they could perform. Not sure why the phone lines "jam up" exactly, don't you have a hierarchical management structure for this sort of thing? Or do 1000 pilots all report to one person?

They've got like 1000 planes and like 100-150 destinations, it's not the traveling sales man problem, an optimal plan isn't needed now so much as a functional one.

Of course, it's easy to bitch about it not being hard when I've never seen the code. Maybe is also tracks hours and does payroll and a dozen other functions.

Re:How Southwest Airlines melted down by lbotos on Wednesday December 28, 2022 @04:36PM (#34166311)
I suspect crews timing out also play a big factor into this planning challenge:

https://flyingbynumbers.com/cabin-crew-duty-timeouts/

Re:How Southwest Airlines melted down by landemva on Wednesday December 28, 2022 @05:48PM (#34167084)
The safety rules for work hours have been known for ages, so this is not a new challenge for management.
Re:How Southwest Airlines melted down by TheCondor on Wednesday December 28, 2022 @07:23PM (#34167893)
Yes, but this has to be less of a problem when they've been canceling tons of flights, right?
Re:How Southwest Airlines melted down by toast0 on Thursday December 29, 2022 @01:04AM (#34170026)
Depends on the specifics of the duty time rules and how the flights were canceled. If the crew showed up and started getting the flight ready, but the flight was cancelled before takeoff, that may count as duty hours and could reduce their availability in the near term.

If the duty time rules only count time after takeoff, or only time after doors are closed, those both offer a different set of times.

Re:How Southwest Airlines melted down by masklinn on Wednesday December 28, 2022 @04:43PM (#34166383)
> Is resumption difficult

The resumption itself.

> Not sure why the phone lines "jam up" exactly, don't you have a hierarchical management structure for this sort of thing? Or do 1000 pilots all report to one person?

It's not a problem of management (well it is a problem of 15 years of management fuckup which is the root cause), it's a problem of the scheduling system needing manual updates when things didn't go to plan.

At this point it's completely screwed, so it needs to be completely reset, as in the entire scheduling system needs to be reconfigured from empty, more or less.

And because SouthWest operates entirely on point to point, the same cascading properties which led to its complete collapse mean it needs to restart in a somewhat synchronised manner, otherwise you fly 3 planes, there's no followup, and you're hosed again.

> They've got like 1000 planes and like 100-150 destinations, it's not the traveling sales man problem, an optimal plan isn't needed now so much as a functional one.

The system completely lost track of crews, so all of them need to be relocated, their work cycles reworked, flight slots need to be reallocated, flights need to be re-encoded.

Re:How Southwest Airlines melted down by flandish on Wednesday December 28, 2022 @06:34PM (#34167489)
Here is where we should take note of how things were resolved (or are in the process of) and note where “throw more money at it” (fly inefficient routes, farm out fares, etc) would have also helped.

Then next earnings release - when they post shareholders profits - that’s exactly how much they were willing to hold back to fix this, trading profit for people stuck for days.

Re:How Southwest Airlines melted down by smugma on Wednesday December 28, 2022 @08:24PM (#34168367)
Good point. I think it will be smart for them to post a record loss for this quarter.

Accrue a few billion for refunds, depreciate your software to zero, and start to set aside a few billion for new cloud whatever.

Basically, take this as a huge loss and communicate that you’ve lost a lot of goodwill and trying to make passengers whole is #1 priority (after safety, of course).

Re:How Southwest Airlines melted down by philwelch on Thursday December 29, 2022 @02:47AM (#34170471)
Airlines are extremely low-margin businesses, discount airlines like SWA doubly so. “Posting a record loss” of this magnitude isn’t something that can be afforded. What you’re talking about is likely to turn into some form of bankruptcy. Maybe it’s a relatively minor Chapter 11 where all those refunds get written down by the bankruptcy court, or maybe SWA goes the way of TWA and Pan-Am.
Re:How Southwest Airlines melted down by scarface74 on Thursday December 29, 2022 @06:40AM (#34171669)
Airlines make a lot of their money from loyalty programs

https://www.forbes.com/sites/advisor/2020/07/15/how-airlines...

And loyalty programs are bigger business for the higher end airlines that cater to business travels and the more affluent.

On top of that, the top 3 airlines make a lot of money off a first class and business travelers who are using other people’s money.

Re:How Southwest Airlines melted down by philwelch on Thursday December 29, 2022 @10:02AM (#34173265)
You just listed a bunch of revenue sources SWA doesn’t have.
Re:How Southwest Airlines melted down by scarface74 on Thursday December 29, 2022 @10:15AM (#34173434)
That’s the point…
Re:How Southwest Airlines melted down by philwelch on Thursday December 29, 2022 @10:19AM (#34173475)
Gotcha.
Re:How Southwest Airlines melted down by cmmeur01 on Thursday December 29, 2022 @08:26AM (#34172367)
Or maybe their lobbyists get on the phones and secure them a nice bailout, since they’re so “essential” as we’ve just seen.
Re:How Southwest Airlines melted down by chasd00 on Thursday December 29, 2022 @08:52AM (#34172582)
Replying to a sibling comment. There will never be a bailout for SWA. Too small, too many competitors and based in a red state with a blue administration in power.
Re:How Southwest Airlines melted down by smugma on Thursday December 29, 2022 @02:52PM (#34177080)
They lost $300M in Q1 and made $800M in Q2. They can take another $300M loss (1B?) in the short term to mitigate reputational damage and legislation.
Re:How Southwest Airlines melted down by panarky on Sunday January 01, 2023 @06:24PM (#34212240)
In the last fiscal quarter pre-pandemic, they had operating income of $3 billion on revenue of $22 billion, or 13%. This is not a low-margin business.

If they have operating income of eight to ten billion a year, spending a third of a billion to upgrade systems seems like it would have been a reasonable investment instead of buying back shares and increasing the dividend.

Re:How Southwest Airlines melted down by TimTheTinker on Wednesday December 28, 2022 @06:22PM (#34167390)
It's essentially an optimizing constraint solving problem. As far as I know, there are 2 main known approaches to this class of problem:

- use a tree traversal algorithm with a lot of built-in optimizations and pruning. Google OR-Tools is one well-known solver.

- use a set of meta-heuristics (Tabu search, simulated annealing, etc.) against arbitrary predicate expressions. The best example of this kind of silver is OptaPlanner.

The advantage of tree traversal is that it is exhaustive and guaranteed to be optimal; but it requires a fixed computing budget for a given set of constraints. Given a large compute cluster it's ideal, but on individual machines/servers these solutions tend to take hours or days. Southwest's system would likely require a significant supercomputer to run its scheduling through an exhaustive optimizing constraint solver, since it would have to re-run the entire solution (or significant numbers of subtrees) when any variable changes (which happens likely several times per second).

Meta-heuristics are more flexible and allow for all sorts of interesting, convenient features that can be very helpful when your compute budget is less than a Tiobe-500 supercomputer. They offer time-constrained solving, real-time monitoring of a solution in progress (you can watch it get better as more of the solution space is searched), and over-constrained solving (where the requested solution is impossible, but we need to make a best-effort attempt).

Re:How Southwest Airlines melted down by geethree on Thursday December 29, 2022 @04:44AM (#34171055)
Other paid constraint solvers are Gurobi and IBM’s Cplex. I only have experience with the later - scheduling giant chemistry and biology experiences into a robotic factory - there are a ton of foot guns in this space.
Re:How Southwest Airlines melted down by 0xfaded on Thursday December 29, 2022 @06:53AM (#34171743)
IBM has ILOG as well which is likely a better match for scheduling problems.

Fun Fact: google-ortools was written by the same people that wrote ILOG. You used to have to "bring your own search" when using ortools, which I used to assume was to avoid conflict with IBM. I haven't used it in a while, but looks like there's a one-size-fits-all search algorithm now. I'd be curious to try it sometime.

Re:How Southwest Airlines melted down by ge0ffrey on Tuesday January 03, 2023 @06:07AM (#34230096)
OptaPlanner (Java) or OptaPy (Python) doesn't need any other installations. Just add a dependency in Maven/Gradle (Java) or pip install it (Python).
Re:How Southwest Airlines melted down by scarface74 on Thursday December 29, 2022 @05:02AM (#34171129)
I’m by no means an expert in this area. But from my experience writing field service software in another life, even on a city wide scale when you may have 20 people and around 200 stops it got tedious when the system went down and you had to manually schedule it.

I use to write field service software for ruggedized Windows CE devices.