News for hackers. Stuff that matters.
Re:Killed by a Machine: The Therac-25 by notalaser on Monday August 01, 2016 @10:52AM (#12202789)
Purely hardware-controlled vs. purely software-controlled devices are somewhat difficult to evaluate (partly because purely X-controlled is somewhat of a rarity).

The "trick", or rather, the true test of a design when it comes to this is judiciously balancing the advantages that each of these categories bring you. You generally want to use "hardware" means in order to nullify the important risks of programmable logic (i.e. the major consequences of a blunder in the logic that was programmed on the device), and use programmable logic for those sections that require flexibility, ease of programming, potential fixes or enhancements and so on.

Real-life example from a device I worked on: the linear actuator (that was essentially sticking needles into people's brains) had a physical barrier past the safe distance limit. Literally a big hunk of metal that the actuator could not be moved past.

Of course, there was a limit in software as well (the software would refuse to move the motor past the safe limit). However, with the physical barrier in place, it protected the motor more than anything else.

It's worth noting that design decisions in these fields are done not so much based on the amount of brokenness in existing implementations (to put it bluntly, there's about as much broken hardware as there is broken software), but based on the amount of risk that a solution introduces. Many regulating bodies (e.g. FDA) will classify your device in a higher risk class if there's programmable logic in it, simply because software tends to be harder to write and test reliably (not only because of bad programmers, but also because of a lack of standards, or at least consensus and metrics on testing).

Edit: I can't quote numbers right now; there is data that shows the risks involved in purely software-based approaches, and it's easy to see why even in the example above. A software-enforced limit on the distance of movement can fail due to a bunch of reasons, not all of them bugs. Bugs are one thing, but the system could also fail due to a broken connection between the motor driver and the CPU, due to a bug in the hardware implementation of the motor driver itself, due to glitches on the bus and so on.

There are software mitigations for all of these cases, too (e.g. you continuously monitor the position of the motor; if the motor still moves after you tried to stop it, you reset the system, and the hardware is wired so that all power to the motor is cut when you come out of reset), but nothing is as efficient as making sure that the motor just won't be able to move the load past a certain distance by placing a physical barrier in its way. By an uncanny chain of events, maybe all the software mitigations can fail; nothing, however, can make a big hunk of metal disappear into thin air.

Re:Killed by a Machine: The Therac-25 by shabble on Monday August 01, 2016 @12:24PM (#12203674)
Do you have a 'big hunk of barrier presence' monitor though? Otherwise, when someone removes the physical stop during maintenance and forgets to reinstall it, and nobody notices for a long time because the software safeties are all working as intended, until...!

Obviously, you can't go down the rabbithole forever, and this is a bit tongue-in-cheek, but IIRC similar circumstances have occurred.

The San Salvador medical irradiation facility accident[1] is largely a tale of gradual failure of safety features and interlocks, combined with the misunderstanding that they were still adequate or safe, until they weren't.

[1] http://www-pub.iaea.org/MTCD/publications/PDF/Pub847_web.pdf

Re:Killed by a Machine: The Therac-25 by pessimizer on Monday August 01, 2016 @01:45PM (#12204391)
> when someone removes the physical stop during maintenance and forgets to reinstall it

That wouldn't be seen as the fault of the device. Software bugs will still show up when the product is used as directed. Ideally, though, the device wouldn't be able to be used without the physical stop.

Re:Killed by a Machine: The Therac-25 by notalaser on Monday August 01, 2016 @02:40PM (#12204817)
Indeed, there is no way to remove the physical stop.

Furthermore, the design process is shaped so that it leads to such designs. If the first iteration of the design had included a removable physical stop, it would have been caught in the risk analysis. Most mission-critical fields have standards that specify this sort of stuff (IEC 60601 for medical devices, DO-254 and DO-178C for avionics etc.)

When designing this kind of devices, whether it's the fault of the device or not is not too relevant in general. You have to mitigate any kind of unacceptable risk (i.e. things that lead to injury or death).

There are certain common-sense exceptions here. For instance, the device isn't expected to operate properly outside its specified operation conditions, but you have to clearly state what those conditions are (altitude, temperature, humidity, presence or absence of liquids etc.) and put warning labels regarding their breach in the manual). Similarly, the level of mitigation is often specified by standards. E.g. for medical devices, IEC 60601 specifies insulation requirements that will protect against the kind of shocks you could get from a faulty network, but not if the device is struck by lightning or strapped to an electric chair. IEC 61010 (Safety requirements for electrical equipment for measurement, control and laboratory use) similarly includes provisions for the kind of protection you would need in equipment that falls under a specific type of use (e.g. here on Earth, not up there in space).

Re:Killed by a Machine: The Therac-25 by notalaser on Monday August 01, 2016 @02:23PM (#12204689)
There's no way to remove it, the big hunk of metal is part of the actuator's frame. Basically, the only way to remove it is to cut it off with a chainsaw :-D.
Re:Killed by a Machine: The Therac-25 by jacquesm on Monday August 01, 2016 @03:44PM (#12205299)
> Basically, the only way to remove it is to cut it off with a chainsaw

Chainsaws cut wood, I think you meant an angle grinder or a blow-torch.

Re:Killed by a Machine: The Therac-25 by honkhonkpants on Monday August 01, 2016 @03:44PM (#12205295)
Amazing that the machine involved in the San Salvador incident was made by the same company who made the Therac. Seems as though they don't have the knack for making foolproof systems.

Apparently the organization continues to exist under the name "Nordion".

Re:Killed by a Machine: The Therac-25 by hatsunearu on Monday August 01, 2016 @04:05PM (#12205460)
I see it as just more checks and balances. Hardware blocks and software blocks have vastly different failure scenarios which complement each other well.