The problem with software, which makes it so unlike hardware, is that we can use so much of it, the diversity is huge.
Everything is software now. So much firmware (which is software), and so much hardware that cannot run without software. Your refrigerator, your dishwasher, your car, airliners -- the trend is for everything to be software now.
It is easy to build a finite state machine for a dishwasher that is fairly reliable. Likewise, consumer jet code that hits safety considerations is heavily tested and audited...and the FAA cares about it whenever it causes a plane to crash, leading to a lot of regulations on how the code is vetted.
(Although they still do their best to make the software reliable, they do not bet lives on its reliability.)
Safe Systems from Unreliable Parts https://www.digitalmars.com/articles/b39.html
Designing Safe Software Systems part 2 https://www.digitalmars.com/articles/b40.html
It's about a guy doing similar things, with model rocketry.
It seems though that the resiliency is all due to field tests/ spectacular failure in the latter. (It doesn't detract from the project at all though).
Even when it was new, the AP-101 was part of a family of flight computers that NASA had already been using, and the same is true for the HAL/S compiler used for most of the code.
If you want to explore how it all worked, this SE thread contains some great information and also serves as a good launching point for further research:
https://space.stackexchange.com/questions/19006/what-operati...
In software, you can poke/prod away at the running application to get it to do something it shouldn't. To me, this is a bit like yanking wires that you know are important to see what happens to plane in mid flight. Another anecdote I use is to image pulling the steering wheel off a car while someone's driving it.
I think modern software testing and security analysis is probably far away better than what the ntsb provides because we've already systems in place to cope with crashes.
Graduated drivers license laws for young drivers, age-21 drinking laws, smart airbag technology, rear high-mounted brake lights, commercial drivers licenses, and improved school bus construction standards all came about at least partially due to NTSB recommendations.
How would you solve that? The FAA can issue all sorts of regulations requiring rest breaks, fatigue detection, etc. -- be sure you're ready for that level of intervention before you ask for this level of investigation.
For instance, how many people ride in the car? What is the average car trip length? Why and when are people driving? Speed profiles, line of site in both car and environment. Does traffic all flow one way, can a car ever land in incoming traffic if it loses control, are traffic barriers designed to bring an out of control car safely to rest with good line of sight to the accident? Are features keeping drivers alert or distracting them?
The difference between lots of fender benders with a bit of panel beating and constant fatalities can come down to multiple causes, many outside of engineering a safe car.
There's no good reason not to, for example, have every car use a breathalyzer to start the engine -- those driving drunk on occasion would protect themselves from DUIs, lives would be saved, etc. Or to implement other, similar deep regulations. But the ideology around cars ("freedom"), vs. planes ("safety"), is too different to allow for such levels of intervention.
The way they are implemented now would be similar to only installing seatbelts in cars where the driver has already been in an accident. Similar sentiment was around when seatbelt laws were put in place, "I'm a good driver." but the seatbelt protects you from yourself crashing as well as when someone else crashes into you!
In the same way, breathalysers (ignition interlocks) for everyone wouldn't be "for you" as much as for everyone to prevent them from drinking and crashing into you.
I hadn't thought about widespread ignition interlocks, and obviously there would be a lot of public pushback, and it would be quite intrusive, but I could see it helping reduce traffic fatalities dramatically.
What I would support is a further reduction in the allowed BAC, and stricter enforcement of traffic laws in general.
Almost all crashes are ultimately caused by driver error. Doing FAA/NTSB-style investigation on them would be pointless, because that's exactly what they'd find: people were driving recklessly, inattentively, etc. And there's nothing that can be done about that, because we as a society absolutely refuse to have serious driver training and any standards for driver conduct. We can't even agree on whether the left lane is for passing or not! Try driving in Germany sometime and you'll find that what we accept from drivers here in the US is really quite awful.
1. Data Breach disclosures 2. Reporting/Public statements if the vulnerability or path that caused the breach has been fixed.
Note, this might seem a little weird (like "of course we've fixed the breach!") but often data breaches aren't detected for _months_ after they occur. There's plenty of companies who didn't know they were breached until they were on HaveIBeenPwned.
It's not NTSB level thoroughness, but I think it's a healthy start in that direction.
Gun regulations are a great example of good intentions, bad implementation due in part to ignorance, and industry resistance/interference. Another is PCI which is a self regulation that's often to vague or specific to be useful.
The first commercial aviation company was formed in 1909 (https://en.wikipedia.org/wiki/Airline#History), the FAA was founded in 1958 and the NTSB in 1967....
At some point the "we're in our infancy" stage has to end.
If planes were as reliable as software they would be the leading cause of death in the world.
What we don’t know is how to write reliable software while also delivering an MVP for antsy VCs and shipping feature updates every week.
And that’s fine. Not all software needs extreme reliability. The problem is that a lot of software gets categorized as “doesn’t need it” when it really shouldn’t be, like all those systems holding sensitive personal information.
I’m not sure what the right balance is - the more standards and compliance there are, the more innovation suffers. On the other hand, without standards or legal responsibility, it makes sense for companies to play fast and loose with security, privacy, and stability in favor of features - a consumer is going to pick a better product based on UX, features, and cost, not based on security they have no way of judging.
You'd better stay away from any company that requires FDA approval. Believe it or not, every change is signed. By you. By the FDA compliance person. Etc etc. I'll give you two guesses as to how they decide who goes to prison if God forbid something in the software is shown to have caused a fatality? Or, worse, a series of fatalities?
That sort of regulation happens even in the software industry. It just depends on the purpose of the software.
Source: Worked in FDA-regulated software for six years, at two different places.
Would you mind just telling us instead? Thanks.
The whole process from requirements, specifications, high and low level design, implementation, validation and verification and the rest of the lifecycle requires stringent oversight, including documentation and signoffs. The signatures on those documents have legal meaning and accountability for the engineers who did the analysis and review at each stage.
...in the United States. I definitely recall watching multiple episodes of Air Disasters [1] where surviving pilots in crashes in other countries that killed passengers went to jail for things that in the US would have at most resulted in a suspended license and demotion or termination from their airline.
For example, Air France Flight 296 in 1988. Another example is Gol Transportes Aéreos Flight 1907 in 2006 in Brazil.
[1] AKA Air Emergency, Mayday, or Air Crash Investigation, depending on what country you watch in and what channel it is on.
We know some very good ways to build reliable software. It's just that most industries can't / don't want to pay the costs associated with doing so.
The don't-know / don't-want distinction is functionally irrelevant, but I have a minor tick about it because asserting the former is often used to avoid admitting to the later.
Legacy IBM and ATT, for all their cluster%_$^!s, were amazing at engineering reliable systems (software and hardware). Maybe that's not practical nowadays, outside of heavily regulated industries like military / aerospace, because markets have greater numbers of competitors. But we do know how.
An accurate and modern truth would probably be "We don't know how to quickly build reliable software at low cost."
And so you’re right, maybe my statement is a little bit of a simplification, but not by much. The amount of man hours it would take to prove that the Linux kernel had no bugs according to some specification would require an absolutely prohibitive amount of time. As such formal verification is so prohibitive that only the most trivial systems could ever be proved.
This is pretty similar to saying that we don’t know how to build bug-free code. We need a paradigm change for things to get better, and I’m not sure it will happen or if it’s even possible.
Would that be a better world?
Sure, I hate it when my word processor crashes. On the other hand, I really like having a word processor at all. It has value for me, even in a somewhat buggy state, more value than a typewriter does. Would I give up having a word processor in order to have, say, a much less buggy OS? I don't think I would.
I hate the bugs. But the perfect may be the enemy of the "good enough to get real work done".
It goes without saying that if we had more time, there would be fewer bugs.
But there are also things (tooling, automation, code development processes) that can decrease the amount of bugs without ballooning development time.
Things that decrease bugs_created per unit_of_development_time.
Automated test suites with good coverage, or KASAN [1, mentioned on here recently]. Code scanners, languages that prohibit unsafe development operations or guard them more closely, and automated fuzzing too.
[1] https://www.kernel.org/doc/html/v4.14/dev-tools/kasan.html
You don’t need anything fancy – but you do need to understand your requirements and thoroughly simulate and verify the intended behavior.
I worked as a mechanic years ago, and I feel the same way about cars as you (and I) do about software. Everything is duct taped together and it's a miracle any of it works as well as it does.
Safety regulations and tests are gamed to all hell, reliability almost always seems to come from an evolutionary method of "just keep using what works and change what doesn't" with hastily tacked on explanations as to why it works, and marketing is so divorced from reality that people assume significantly more is happening in the car than really is.
Now I don't know anything about civil engineering or aricraft, but I have a hunch it's not as well put together as it may seem at first glance.
I still think that software is behind those other fields, and I'm honestly still on the fence if regulations would significantly help or would just slow things down, but it's absolutely the "wild west" in software.
I work as a developer and everyday I feel like this isn't the right the way to do it. I'm not sure what the right way is, but being a software developer I probably have less faith in it then the general population. Something just feels wrong about it, how we do things..
Of course for most commercial software correctness is so irrelevant that it's left undefined. All that matters is pleasantness, that is to say does it generally please the user, perhaps by attracting customers or investors.
It's clear in IT we always choose faster and cheaper. No one went to jail for the Equifax hack except that one low level manager for insider trading.
There are large sectors in this industry where rigorous engineering effort isn't really applicable. It seems to me, though, that this has effectively been used as an excuse to avoid any genuine rigor. It isn't helpful that so many programmers have such huge egos.
Every time there's a data breach, an agency would investigate with the same thoroughness they do in aviation.
Then, National Geographic could have a spin off show called Data Breach Investigation (their aviation show is called Air Crash Investigation in the UK, I think it's called different things in different Countries.)
A man can dream...