There's a lot of negativity and disdain in this thread, but something is being overlooked: You simply can't apply "startup-style" development to something like air traffic control. This isn't your latest javascript app where you can push an update to fix localization bugs after the client reports an issue.

This is air traffic control, and thousands of lives are at risk every day. This stuff has to work. It has to be bulletproof.

The FAA doesn't refactor existing codebases into Haskell on a whim. A lot of this code and the hardware it runs on has been thoroughly vetted, not to mention battle-tested over decades of use. "Move fast and break stuff" is not an attitude I want FAA-certified gear to be built with.

This makes me wonder where the threads (beyond server scaling) around resiliency on HN are. I'm doing some non-critical embedded programming at the moment (typically do full stack stuff), but embedded and critical systems are much different than the typical audience here....

It's easier, flashier, and more immediately gratifying to slap together a new javascript framework.

Embedded is hard.

Fault-tolerant is hard.

Debugging is hard and also very tedious.

It's way easier to build a web company than something that involves difficult programming, may well involve hardware, and probably doesn't have the upside potential that a web thing does. That's why there isn't as much of that kind of thing.

That said, I love to read about embedded stuff, because some of those guys are really sharp. I'm also a user and fan of Erlang.

I think the negativity is warranted. There is too much bureaucratic overhead and even it wasn't purely bureaucratic it should not take a decade to design a resilient and fault-tolerant software/hardware system. Margaret Hamilton (http://en.wikipedia.org/wiki/Margaret_Hamilton_%28scientist%...) might have a few things to say on the matter.

When I build a tool shed in the woods in cottage/lake country, I can do things a lot more casually than someone building a skyscraper in Manhattan can. I probably don't even need a permit.

At no point did I say that the process behind NextGen is perfect.

What would you call sending people to the moon? A tool shed in the woods or something else entirely? It's not like there isn't documented historical precedent for building such complicated systems. That's what the link to Margaret Hamilton was supposed to illuminate but I guess I should have been more explicit.

> What would you call sending people to the moon?

A huge dangerous experiment. Out of 12 Apollo missions, 2 ended in disaster. The Apollo 1 fire killed 3 people, and we all know the story of Apollo 13. I wouldn't call the space program an acceptable comparison for how our commercial aviation system should work.

I suspect that the requirements for the Apollo software are significantly simpler than those needed for air traffic control.

Maybe, maybe not. The general principles should still be applicable.

I worked on the shuttle program for a while and was debugging some fortran code that called honeywell assembly as part of a rewrite of the DOLILU program. It calculated stress on the shuttles frame based on winds aloft data collected by balloon on launch day.

The code passed in a jump parameter and the assembly jumped back to the appropriate fortran label. Over the years, all the exits except one had been coded out. I asked my boss if we could just make the fortran code jump directly from point A to point B since that's all the assembly did. He answered my 6 months contract would be done before the change made it through committee.

The thing about process is people rarely lessen or remove it once it's in place. But it's possible to do a classic waterfall project on time and on budget, look at any bank release of ATM software. Of course losing an orbiter or losing a plane isn't the same class of problem as losing a transaction, but it is possible to do the projects properly.

The fact the Obama administration had to bring in outside help to rescue their most recent major rollout (Obamacare insurance info websites) tells you there's no incentive to get it right or on time in these contracts. You just keep working and the bills keep getting paid.

These contracts need real oversight and financial penalties for failure to meet deadlines. Assuming of course they also include some iron-clad agreements about requirements not changing.

I'm not saying your wrong, but shuttle software and even ATMs are different from Obamacare. The former has specific operational requirements that are relatively clear. Health care by contrast is a clusterfuck of legislation piled on top a multitude of tangential legacy systems. I don't doubt that the talent to get it right exists, but it requires more juice than just an engineer saying "this thing won't fly". The problem is that legislation is itself code, but it is written by people who have no nuanced knowledge of the downstream implications, and perhaps more importantly, no incentive to ever refactor.

Agree. Physics doesn't lie. But Politicians lie (or don't know what they are talking about OR pretend to not know what they are talking about).

Or both at the same time. They answer the question they wish you had asked.

http://jezebel.com/male-politician-thinks-that-your-vagina-a...

Obamacare and shuttle software are completely different political beasts.

Obamacare is a standard government contract. The contractor is piling money as fast as possible into today so that when the government pulls the plug tomorrow he's already made a profit.

The shuttle software is a safety critical system with a defined budget and schedule. There is no overall incentive to do things immediately and badly as the political fallout for failure will be very severe.

As a pilot myself, I wouldn't be so fast to blame the FAA (as much as we pilots don't like them at times). The FAA is using 40 year old tech for the same reason we (especially general aviation pilots) are flying 40+ year old planes: there used to be tons of money pouring into aviation, and now there's not. It's prohibitively expensive to do just about anything in aviation nowadays.

But that's in large part because of the FAA. Look no further than the LSA category, which never took off because of the onerous regulatory environment. I think it's much more likely the 40 year old tech is a product of risk-aversion than a lack of money pouring into aviation.

> THE UPGRADED SYSTEM WILL GO ONLINE THIS SPRING, FIVE YEARS LATE AND AT LEAST $500M OVER BUDGET.

If they are over budget by half a billion, what was the original budget number?

I completely understand that building such system is complicated, not just in terms of technology but in mainly in terms of coordinating all the moving pieces involved. But at no point, I can fathom how you need to spend close to a billion dollars to implement it. As Milton Friedman famously said, you are least careful when you are spending other's money. (i.e. Government spending taxpayer's money)

It takes close to half a billion dollars just to test it. Stuff has to work right the first time, and every time after that, for 40+ years. They (and by they I mean we) test the human factors of the displays the controllers work at, the communications links between the ground and the air, the changing of routes for thunderstorms or wind changes, the interoperation of communications and navigation systems built across several decades, the algorithms for deconflicting merging aircraft and each type of aircraft's ability to follow the deconflicting instructions... stuff's complicated, yo.

Twitter's worth $38 billion, so I guess ATC for the US should be worth 40 million billion gajillion dollars. Sounds like an extra $500 million is chump change.

"FOR THE FORESEEABLE FUTURE, IF YOU PURCHASE WI-FI IN COACH, YOU'RE PRETTY MUCH BETTER OFF THAN THE PILOT."

So why does ATC not having GPS tracking make the pilot worse off? The pilot does have GPS, and even more precision navigation in certain cases like instrument landings.

If the FAA and the companies they keep hiring weren't so f'n sclerotic, I'd say they need to look at a totally decentralized system that account for autonomous drones, so all positive control flights (non-VFR) can be integrated into the same system; but obviously starting over yet again is a great way to just burn another $5 billion. No actually burning the cash would at least have residual value as potash.

It's unsafe for aviation to be entirely dependent on GPS and self-reporting systems like ADS-B. Planes should still have VORs available, even though they take up a lot of acreage next to airports, the ground should still have primary radar, and aircraft should still have their dumb old AM VHF radios.

Large aircraft typically have at least three completely different navigational sources - VOR/DME, GPS, and inertial guidance. Over- reliance on GPS is dangerous.

NextGen gives the pilots (with "ADS-B in") the same view of the airspace that the controllers get. Right now, they only have a limited view, via the TCAS.

Yes, but WiFi doesn't give me the same view of the airspace the controllers get, or pilots will eventually get. WiFi gets me something like flightware tracking which is a lot slower and more delayed than primary/secondary radar, TCAS or ADS-B.

My point is I think the article's author thought they were making a funny. And it wasn't correct or funny.

Reminds me of 7 years back when I was working at a major power generation company. There was a team working on restarting production of completely obsolete analog control equipment for nuclear reactors. The operator wanted to extend the life of their reactors by a decade or two and it just wasn't worth it for them to have to re-certify a modern system. The existing system had been running for decades without any serious accident, so with stakes that high and red tape that thick, they just went for "if it ain't broke, don't fix it". It was funny in a steampunk / Fallout kind of way.

Seems totally reasonable, provided the system is maintained and operated within design limits.

"If it ain't costing me money in a way I can see on a simple spreadsheet, don't fix it."

There. That's the mature way of maintaining systems. That's how grown-ups approach the world.

One of the core components of the NextGen system, ADS-B, is already mandatory for airlines in some countries (such Australia). Its how flightradar24 and other sites manage to get real time info.

http://www.flightradar24.com/about

http://en.wikipedia.org/wiki/Automatic_dependent_surveillanc...

"Lockheed Martin began developing the software for it in 2002"

I see your problem right there.

I had worked on this a few years back, and I quit to start a startup. Sorry. (Not sorry.)

The problem is that most people confuse bureaucracy with process and structure. Those are not the same thing. It is possible to have process and structure without bureaucracy.

ugh: "THE UPGRADED SYSTEM WILL GO ONLINE THIS SPRING, FIVE YEARS LATE AND AT LEAST $500M OVER BUDGET." #

Before Lockhead Martin's work trying to modernize, IBM had a billions dollar contract in the 80's.

http://sebokwiki.org/wiki/FAA_Advanced_Automation_System_%28...

http://articles.latimes.com/1988-08-11/business/fi-288_1_air...

Basically, none of these huge companies want to do this kind of work unless the pay is obscene. And then once it's obscene, the companies all fight over it and throw each other under the bus. Court battles aren't even the half of it. They hire lobbyists, contribute to campaigns, and every 2-8 years there's some amount of government shuffling. So nothing is actually ever very consistent within the government. New politicians, new bureaucrats, new policies, new ideas (they think anyway), and they basically have been scrapping previously done work and starting over multiple times. And it's been longer than 40 years.

Perversely, running on vacuum tubes makes the old system a more security hardened system than what we're likely to end up with. We should hope the NSA has hacked the FAA and ATC as a defensive measure.

There's a lot that goes into ICAO certification of telecom technologies. You can't just use contention-based protocols, you need some sort of deterministic bound on latency and performance and availability requirements are insane. Different worlds with different requirements.

Every time I see that critical systems like these are running on very old tech I am wondering whether or not the programmers that built these a couple of decades ago are way more capable than the ones from today.

Did I miss it or do they not mention what tech it is?

Are they talking about this? http://en.wikipedia.org/wiki/IBM_9020

They did not mention it. I do believe that is the technology.

The current system is 40 years old, but the new system will be almost 20 years old by the time it's fully deployed (2020).