> Actually testing was built into the software. When it came up it would talk to the physical parts to make sure everything was communicating ok before it could start running.

I've been wondering about this for a while. We tend to run unit tests, integration tests, whatever tests, while the software is in development. However, once it is in "production" (for whatever definition of production), usually no tests are performed. At most, there's some sort of metrics and server monitoring, but nothing in the actual software.

It's a work of fiction, but Star Trek has these "diagnostics" that are run, with several levels of correctness checks, for pretty much everything. In your typical app, it could be useful to ask it to "run diagnostics", to see if everything was still performing as expected.

Early in my career I worked on military flight data recorders, including the development of the software for the F-22's "black box". Those systems have SBIT, IBIT, PBIT and MBIT sub-systems were BIT is "built in test" and S = startup, I = initiated, P = periodic and M = maintenance. I remember making the Star Trek diagnostic joke myself when I was assigned the SBIT work.

Each BIT does varying level of testing based on it's runtime budget but there are a lot of very basic tests that don't make much sense until you see your first field report of some register bit "sticking". Its much better to ground a plane that can't add 1+1 than to find that out in mid-flight.

An F-22 had a FCS failure and didn't realize takeoff. The pilot didn't do a IBIT on the FCS as required, therefore wasn't aware that all 3 rate sensors latched up. The jet was uncontrollable once it left the ground, and the pilot ejected safely. [0]

As I recall, they modified the sensors to avoid latch up (extra pullup resister) and updated the FCS software to provide a warning if all 3 sensors return zero output. Even though this wasn't really an error in the FCS software, It could be argued that it failed to detect an erroneous input from 3 redundant sensors that latched up for the same reason.

This is a classic example of a pilot who misunderstood the pre-takeoff checklist procedure and cost the USAF a $325m aircraft.

[0] http://usaf.aib.law.af.mil/ExecSum2005/F-22A_20Dec04.pdf

To be fair this seems to be more of a classic example of a documentation or training problem. Right from your link:

> During the mishap sequence, the MP started engines, perfomled an IBIT, and had a fully functioning Flight Control System. Subsequently, the MP shut down engines to allow maintenance personnel to service the Stored Energy System. During engine shut down, the MA's Auxiliary Power System (APU) was running. The MP believed the APU provided continuous power to the Flight Control System, and therefore another IBIT after engine restart was unnecessary. This belief was based on academic training, technical data system description, and was shared by most F/A-22 personnel interviewed during the investigation.

  To be fair this seems to be more of a classic 
  example of a documentation or training problem.

Maybe - but if you were designing a consumer product, you wouldn't rely on the user following a checklist; the IBIT would run automatically when they turned on the ignition and sound an alarm (or even prevent takeoff) if the vehicle would be uncontrollable.

So you could also classify this as a user interface / design problem.

SBIT & PBITs are the ones that run automatically, with SBIT running automatically at startup and PBIT running on a watchdog. The SBIT time budget and scope is usually much smaller than IBIT so time intensive tests like ones that talk to sensors on the bus aren't present in SBIT. You can think of the stages as SBIT: can I run? IBIT: should I run? PBIT: am I running right?

True. The pilot wasn't the only one who misunderstood the details.

What types of errors would cause one of the tests to fail? Is it mostly testing for hardware errors, or are there any software logic errors that could make it to production, but be caught by one of the tests several months down the road? The only software related items I can think of are edge cases where a built in test is based on real time input. Kind of like running the calculations through multiple independent implementations of the software.

There's actually a decent probability of memory corruption in space applications due to radiation. So in addition to checking communication across busses, application checksums are typically run continuously.

Another thing that's software-related is if you've got a (rare) race condition. For example a data structure that gets corrupted if you have a particular series of nested interrupts.

Now, hopefully your system is set up such that race conditions cannot happen, but good luck with that.

Yup, as far as I remember all the avionics hardware I saw had these types of BITs built in.

Dropwizard (a web application framework/library) incorporates the idea of such health checks, and you can also implement additional ones specific to the application. It is encouraged to run them periodically in production to ensure that the database connection is still up, threads are not stepping on each others' toes, etc.

https://dropwizard.github.io/dropwizard/manual/core.html#hea...

That uses the metrics library by the same author. Very handy stuff in there for development and production.

https://dropwizard.github.io/metrics/3.1.0/

Even while running the radar would keep track of communications between the parts and make sure things were still ok. The system needed messages periodically from the external components and vice versa to make sure things were ok. There were status messages sent around too. And a display of how things were doing. Its been a while, but I remember some of the things.

You could command the external things to run diagnostics and report back. Part of this was it makes fixing things easier (like your car computer's diagnostics), part of it was required so we can figure out where things weren't working optimally.

For example if a motor was running its controller computer didn't hear from the main system in X seconds, it would just stop, send a message about what it was doing and then wait for an instruction. Presumably this was to prevent all heck from breaking loose if the main system went down or was not responsive.

This was all spelled out in long requirements documents.

I wonder sometime if our cars controlling computers are doing this...

Cars often disable any system capable of interfering with the wheels at the least sign of issue. Insufficient seal on the fuel-cap? Disable ABS, TCS, etc.

That's not correct. Cars will generally have something that's often called a 'limp home' mode. I saw a car that's PCM (powertrain control module) that failed some internal test, so it fell back to a basic mode where the engine wouldn't rev over about 2000RPM, and I'm sure all the emissions systems went into a basic fail-safe mode, where things like the fuel delivery goes into a hard-coded mode, instead of using feedback from the oxygen sensors to tune the fuel delivery. But a loose fuel cap doesn't disable the ABS or TCS. Even a single faulty wheel speed sensor doesn't have to disable the entire ABS, it can still independently assess each wheel, determine if a lock-up is imminent and modulate the brakes for that wheel.

That's neat. The counterpart for spacecraft is "safe mode" (http://en.wikipedia.org/wiki/Safe_mode_(spacecraft) ). The priorities are maintaining attitude control, conserving power, and listening to the radio. The science instruments and other bells and whistles are turned off as much as possible.

What really scares me are the ABS and traction control systems that are now becoming mainstream on motorcycles. These systems have accelerometers and gyros, which take in to account, information like lean angle of the bike when calculating the braking force. It certainly is a feat of engineering.

Little about being a biker, programmer, and an aspiring tree-shade mechanic, reassures me about the safety of these systems. Internals of a bike are much more exposed to abuse, bikers are known to take a spanner to their machines, the motorcycle repair workshops are a much more informal industry (at least here in India). What happens to a bike that is not subject to regular maintenance? On the other hand, I have always marvelled at how mechanical systems like motorcycles are usually built with some sort of graceful failure in mind - In a lot of cases a motorcycle will warn you about a faulty component before it fails catastrophically. I assume the people who designed these systems, would have kept that in mind while designing these systems (although stuff like Toyota's unintended acceleration does not inspire confidence).

And what happens in the event of a catastrophic failure:- a car locking up at speed is still dangerous, but there is room for error. If the front tires of your motorcycle locks up at speed, the odds of you walking away from the incident are not high.

Don't get me wrong, these systems DO SAVE more LIVES than they could possibly take away in the long run, But I am still disconcerted by the whole thing.

EDIT: typos

> although stuff like Toyota's unintended acceleration does not inspire confidence

Another thing that worries me are that these same companies are also working on self driving cars.

Worst of it is that everyone pretty much jumped into the race after Google. It also doesn't look like existing solutions work on a real time system. I'm a bit worried about being hit by a car because it was running garbage collection process and did not react quickly enough.

That's not correct. You can leave the fuel cap off and it won't disable ABS or TCS. The only thing that will happen is the "check engine" warning light will come on.

Funny. Just the other day, my gas cap was loose. The traction control light came on along with the check engine light. As far as I could tell, it (traction control) was still operational, however.

Seemed like a funny combination of lights to blink on.

It doesn't necessarily take much work to run integration tests as diagnostics on production, especially if you plan on it in advance. I've had good success with it.

Of course there's a certain level of destructive testing you can't do live, but that you really ought to do on your development system, load testing being a simple example. It behooves the wise developer to keep these quite separated in the code. :)

The project I'm working on (for the Federal government no less, and very much not life and death) involves having test data in the production system and tests that run in the production system. (I was impressed by this when I found out.)

Make sure they're well separated! c.f. http://www.wsj.com/articles/SB100014240527487033765045754919...

The stuff we are doing is so not life and death that nothing like this could happen. E.g. We have fairly insane levels of security for information that is inherently public.

That would be called reliability testing and it's why Netflix has chaos monkey to throw a wrench into things.

Also conformity monkey.

I used to write fire control and fire monitoring software. The whole idea of live testing is built into the ethos of such systems. In fire alarms and control the only tests that matter are the ones performed in a production environment.

A lot of ppl get this backwards, that a stable program should never "crash". While it's actually the opposite, it should throw errors at every opportunity to do so.

The errors should then be logged and the program should be restarted by a watcher process.

Here's an example on how you can both log errors and e-mail them if a process crash, using a startup script (Linux, Ubuntu):

  exec sudo -u user /bin_location /program_path 2>&1 >>/log_path | tee -a /error_log_path | mail [email protected] -s email_subject

This is how Erlang (for example) gets its reputation of being "nine-nines" capable (i.e. capable of 99.9999999% uptime, or downtime on the order of milliseconds per year). Erlang (and Elixir and LFE) software following the OTP framework is usually ordered into "supervision trees" - layer upon layer of Erlang processes managing other Erlang processes in turn managing other Erlang processes, all potentially distributed across multiple Erlang VM (nowadays BEAM) instances.

A watchdog pattern just splits the program into several processes, the program as a whole still never crashes.

If an error is caught and handled, calling it a crash seems disingenuous.

One mistake that ppl do is they wrap their code around a try ... catch, where it's better to throw an error and exit. If there's an error in one place, chances are there are also errors elsewhere, so it's better to restart the program instead of continue with a bad state.

When the error gets thrown in your face, there's a higher chance that it gets fixed.

But this also have its setbacks. Loosing the whole state can be really bad.

I'm really note sure why you think catching an error in a separate process is somehow superior to catching it in a higher scope

I think it depends on the kind of error. If it is a "bug-detected" error (null-pointer dereference, out-of-bounds, divide-by-zero, out-of-memory, etc.), you better restart the program since you're in an unstable state. If it is a "domain-specific" error (connection lost, robot could not reach its destination, battery low, etc.), you better deal with it as soon as possible.

> Reliability was favored over other things (for example recursion was discouraged).

This sounds really strange to me. So may I ask why? I find that recursion - most of the time - helps shorten and clarify the code. Also, doesn't recursion make induction proofs trivial?

It's a lot harder to reason about memory constraints on recursive programs.

The clearness and correctness of the code often ignores the possibility for stack overflow. Most naive implementations of DFS will hit the stack limit given trees that are all one long path from a single root to a single leaf.

Further to your point, here's the guideline, and rationale, from Gerard Holzmann's document on recommended coding practices for C at NASA/JPL:

1. Rule: Restrict all code to very simple control flow constructs – do not use goto statements, setjmp or longjmp constructs, and direct or indirect recursion.

Rationale: Simpler control flow translates into stronger capabilities for verification and often results in improved code clarity. The banishment of recursion is perhaps the biggest surprise here. Without recursion, though, we are guaranteed to have an acyclic function call graph, which can be exploited by code analyzers, and can directly help to prove that all executions that should be bounded are in fact bounded. (Note that this rule does not require that all functions have a single point of return – although this often also simplifies control flow. There are enough cases, though, where an early error return is the simpler solution.)

This is rule 1 of 10, so he apparently feels strongly about "banishing recursion." Gerard was formerly at Bell Labs and is also a fellow of the ACM and a member of the NAE.

It's also important to note that these rules are made for critical control systems that tend to be low level. The cost benefit trade offs aren't going to be the same as in typical business software.

It's not a bad question, but approaching it from a CS perspective will cause you to blow your foot off--because it's not about code length or code clarity, it's about safety (which is orthogonal). Your stack's of a finite length, and eventually will grow into the heap unless your system has protections against it.

In most systems lots of really important stuff is allocated at the bottom of the heap. It's very easy for a clobbered global flag (yes, hissss, globals, these are very constrained computers we're talking about here) to cause a system to have its shit get real at an alarming rate.

The issue is largely that unbounded recursion is quite easy to do accidentally (in many of the languages that where used in the past) with the resulting stack smashing causing issues.

Also many of these systems where hard real time as in "if we don't respond in under 30ms something expensive goes bang" and again recursion can cause problems with that, lots of these systems are interrupt driven and have no garbage collection or threading so you can't just pre-empt them in that event since by the time you spot the problem you blew through your deadline and something went bang.

Yup. We had some processes running with their own cpu with interrupts turned off, so if the process went weird it meant reboot (as we discovered the hard way one day). So we tried to keep code simple.

On the plus side you had a pretty good idea about how long the max processing would take (and avoid the timeouts and aforementioned "bang"), as the OS couldn't interrupt us. Certain system calls couldn't be made while in what we called "soft real time". Memory allocation was done upfront.

The process control of that system was interesting. You could assign processes to processors or groups of processors and then give those groups a scheduling method. I haven't seen anything like it in the years since I left.

Non recursion is waaay "easier" to troubleshoot finite precision floating point issues and failures of the first part of "be liberal in what you accept and conservative in what you send".

If you assume infinite precision arithmetic and a very friendly environment for inputs, recursion always looks simpler, but by the time you clean it up to handle real world issues, non recursion instead looks simpler.

Its too easy to write recursive end conditions along the lines of "if x == 42" when your helpful floating point routine somehow mysteriously rounded x to 42.00000001 so it'll never equal, or "no (supposedly) UTF-16 encoded string would ever have an odd number of bytes, even though I have no control of the source and the source is known to occasionally be insane" or at least thats how I remember it. I've run into both. Its not funny at the time but in retrospect its usually fairly hilarious.

Personally I think its harder for people to understand concurrency issues WRT recursion, but I'll probably just get flamed for that one. I feel more people have "leveled up" with concurrency and non-recursive code and functional style programming than have leveled up to include recursion in that mix. Imagine two (three?) concurrent recursive algos fighting each other over one data structure.

Every recursive algorithm has an equivalent iterative double. Iterative solutions ARE induction :)

If you are careful about test vectors, you can pseudo-exhaustively prove that an iterative dual to a recursive algorithm is equivalent.

In addition to what others have said here; often recursive calls can also trivially be optimized into loops by the compiler.

This is very convenient in most cases, but could hide the fact that a direct recursive call will overflow your stack.

This isn't apparent and would test out ok until someone makes a small; seemingly insignificant change which the compiler can't do tail call optimization on, and all of a sudden things fail.

tldr version: Because you want (i) an acyclic function call tree of (ii) predictable depth.

That reminds me--a friend was telling me earlier today about a piece of software he was working on that only allowed forward jumps. (It ensures that the program halts.)

It does slightly more than that - it provides a cheap computation of the upper bound of how long it will take.

Yeah, that too. Come to think of it, that was probably more his concern than just halting.

Going out on a limb here, but I don't suppose it was for a superoptimizer?

> An when they built the physical radar, the software worked.

I can only imagine a feeling of satisfaction you and your colleagues felt at this moment.

I've seen some software that does lots of tests on startup. One software I used (ATG) validated all the ORM mappings against the currently configured database on startup by default.