News for hackers. Stuff that matters.
Re:Therac-25 by ygra on Tuesday February 18, 2014 @04:06AM (#7257125)
This was one example given in a very early lecture about software testing and failures in our uni. This (and others) make me hope that I never have that much responsibility over human lives as a developer.
Re:Therac-25 by chiph on Tuesday February 18, 2014 @08:55AM (#7258074)
Unless your code is the most trivial ever, there's still a tiny possibility that it could result in a death.

Game: User goes into epileptic seizure, hits their head and dies.

Social media: User calls for help with it - "Please send help #carcrash #bleeding" and the message is lost.

As an engineer and a responsible human being, you don't ever want this to happen. But it will. You have to minimize the chances of it happening, with the realization that it will never go to 0.0%

The world runs on software today. With all that that implies.

Re:Therac-25 by pjmlp on Tuesday February 18, 2014 @04:35AM (#7257203)
It is this type of failures that drives me to push for software quality and moving away from languages like C.

As well as, consider that liability should be part of the industry as it is in other industries.

Sadly the drive for profit speaks otherwise.

Re:Therac-25 by mortov on Tuesday February 18, 2014 @04:56AM (#7257284)
The issue is not because of the language used. You can have bugs - potentially fatal as this case underlines - in any language.

When I studied the Therac case, there was also a study in the "Killer Robot" - see here : http://www.onlineethics.org/cms/5122.aspx. Well worth a read to understand how software can become dangerous.

Sadly nowadays, a search for Killer Robot turns up all sorts of stories of drones being used to kill people. When I first looked up killer robot a number of years ago, the results were pretty much all about the ethics scenario and not real life people being killed. How society has moved on...

Re:Therac-25 by Piskvorrr on Tuesday February 18, 2014 @05:23AM (#7257359)
Also relevant (and on HN newest as of now) - Toyota accelerator issue: http://embeddedgurus.com/state-space/2014/02/are-we-shooting...
Re:Therac-25 by fnordfnordfnord on Tuesday February 18, 2014 @08:09AM (#7257900)
This one isn't merely an example of a failure of engineering, but also of organizational failure at Toyota, and organizational failure at both the NHTSA and NASA.
Re:Therac-25 by evincarofautumn on Tuesday February 18, 2014 @05:01AM (#7257295)
> You can have bugs - potentially fatal as this case underlines - in any language.

That is undoubtedly true. However, the commonest classes of bugs can be ruled out by languages with more expressive static verification tools—type systems, effect systems, and so on—leaving us with a stronger baseline from which to work.

Re:Therac-25 by pjmlp on Tuesday February 18, 2014 @05:21AM (#7257350)
> The issue is not because of the language used. You can have bugs - potentially fatal as this case underlines - in any language.

This is true, however there are languages which allow for more programming errors than others.

I just mentioned C as one possible example.

The goal should be to have programming languages that reduce programming errors, not that make it easier to shot yourself.

Of course this is only a small step towards better software, as whole system design also plays a big role.

Re:Therac-25 by troels on Tuesday February 18, 2014 @05:22AM (#7257358)
It's explicitly stated that the choice of language was not considered a root cause. And that was even asm - not C

The software was written in assembly language that might require more attention for testing and good design. However the choice of language by itself is not listed as a primary cause in the report.

Re:Therac-25 by pjmlp on Tuesday February 18, 2014 @05:26AM (#7257372)
C was just as example, as it is nothing more than a portable macro assembler.
Re:Therac-25 by ColinWright on Tuesday February 18, 2014 @05:55AM (#7257456)
This is a common mis-conception invariably produced by people who've never written a C compiler. C is not just a portable macro assembler.
Re:Therac-25 by pjmlp on Tuesday February 18, 2014 @06:53AM (#7257611)
> This is a common mis-conception invariably produced by people who've never written a C compiler.

Wrong on that one. I do have a good background in compiler design.

> C is not just a portable macro assembler.

Given what it offers when compared with safer systems programming languages, it is one according to my own definition. :)

Re:Therac-25 by troels on Tuesday February 18, 2014 @05:49AM (#7257437)
As opposed to being written in actual assembler?
Re:Therac-25 by weland on Tuesday February 18, 2014 @05:21AM (#7257349)
This is the bug that occurred:

> The defect was as follows: a one-byte counter in a testing routine frequently overflowed; if an operator provided manual input to the machine at the precise moment that this counter overflowed, the interlock would fail.[3]

Can you describe how exactly a language other than C would have helped with this?

Re:Therac-25 by hrjet on Tuesday February 18, 2014 @05:30AM (#7257383)
A higher level language could have relieved the programmers from low-level details, thus giving them more cognitive space for avoiding logical bugs in other sections of their code.

In general, the more DRY the code, the more mind-space for review / testing.

Re:Therac-25 by JoeAltmaier on Tuesday February 18, 2014 @10:52AM (#7258727)
Remember,this was a PDP11. Don't know what that was? Well, C WAS its high-level language.
Re:Therac-25 by weland on Tuesday February 18, 2014 @05:50AM (#7257443)
> A higher level language could have relieved the programmers from low-level details, thus giving them more cognitive space for avoiding logical bugs in other sections of their code.

This can also be said of choosing simpler design, organizing the code in a better manner, or could have been avoided through more comprehensive testing -- all of which were pointed out by the official report.

Re:Therac-25 by hrjet on Tuesday February 18, 2014 @06:20AM (#7257531)
I am not sure I understand your argument. Using a high-level language doesn't mean one wouldn't follow other best-practices.
Re:Therac-25 by weland on Tuesday February 18, 2014 @08:03AM (#7257881)
Having the bug end up in production could have been avoided simply through adequate testing and verification. I fail to see how a higher-level language helps with that.

The fact that the programmers might have had less stuff to worry about and, therefore, saved their concentration for more critical parts of the code, is hardly direct evidence that the bug could not have been introduced. It can equally well be argued that a higher-level language would have attracted programmers with less experience in mission-critical code, who would have failed to grasp even the possibility of bugs more subtle than this one.

Re:Therac-25 by angersock on Tuesday February 18, 2014 @11:58AM (#7259223)
In general, the more DRY the code, the more mind-space for review / testing.

It's something that would be considered firmware today, written in assembler.

Trying to be DRY in assembly is a really great way of introducing subtle bugs.

Re:Therac-25 by pjmlp on Tuesday February 18, 2014 @05:25AM (#7257367)
Safer system programming languages cause a panic on overflow, instead of happily keep on running.

Unless the programmer has explicitly disabled them, that is.

Crashing is better that corrupt data.

Re:Therac-25 by andyjohnson0 on Tuesday February 18, 2014 @07:09AM (#7257655)
"Crashing is better that corrupt data"

If the crashed program is controlling a device that is directing radiation at a person's body, then crashing is likely to be at least as undesirable as data corruption.

I understand your point about programming language choice, and your comments elsewhere about the undesirability of the C language. But the root causes of the Therac-25 malfunction wasn't technical, they were organisational. Specifically poor code review and code reuse practices. Changing the programming language wouldn't fix them.

Re:Therac-25 by jacquesm on Tuesday February 18, 2014 @11:22AM (#7258956)
> Crashing is better that corrupt data.

In embedded control software of medical systems that are directly affecting the lives of the patients involved crashing is definitely not an option and it is not necessarily better than corrupt data.

What is an option is to very carefully model the permitted states of the machine and then to exhaustively test those states and their transitions to make sure there are no undefined states the machine can be in.

One way to do this is to have a watchdog process (or even an entirely different system) that monitors the state of the machine and that initiates a safe shutdown if any departure of the defined operation is detected.

Re:Therac-25 by ars on Tuesday February 18, 2014 @05:27AM (#7257374)
Kinda curious - which languages panic on overflow?
Re:Therac-25 by pjmlp on Tuesday February 18, 2014 @07:03AM (#7257639)
Ada, here is a small list of validations

http://en.wikibooks.org/wiki/Ada_Programming/Pragmas/Suppres...

Modula-2, a possible documentation link

http://www.excelsior-usa.com/doc/xds/isom203.html#15

Just to cite two examples.

Re:Therac-25 by weland on Tuesday February 18, 2014 @07:32AM (#7257754)
So what happens when the Ada program runs into a failed compiler check? Does it:

* Raise an exception? If so, it's up to the programmer to treat it correctly. There are various ways to detect overflows outside Ada; it's still up to the programmer to treat them correctly. Runtime detection of the overflow (in a hopelessly untested firmware of the Therac-25!) would have brought the matter no closer to resolution.

* Abort execution? If so, is the behaviour of the program defined if execution is aborted? Because undefined behaviour from a control system is quite worse.

Re:Therac-25 by jv2 on Tuesday February 18, 2014 @11:14AM (#7258894)
Normally, raise an Exception; the outermost levels of the program should probably put the machine in a "safe" state if an exception is raised during operation e.g. turn off the radiation beam etc. I would have thought this is infinitely preferable to silently continuing after an error...

SPARK Ada allows the use of source-code annotations to denote information flow in a program, and can vastly reduce the chances of these sorts of problems (at the expense of more programmer time).

Standard Ada compilers can do a reasonable job of detecting some problems at compile-time, but SPARK seems to be the favoured method for safety-critical stuff.

Re:Therac-25 by weland on Tuesday February 18, 2014 @03:04PM (#7260682)
> I would have thought this is infinitely preferable to silently continuing after an error...

It definitely is! However, it still assumes that the exception is safely handled. The flaw in Therac 25's development process was that testing did not reveal this bug. Consequently, the error recovery process could have itself been incorrect and remain uncaught. It's tempting to think that, if error recovery is similar each time, testing it once is enough, but error recovery is as susceptible to being impeded by race conditions as any other part of the code. No uncertainty is removed this way.

It's also worth noting that a value of 0 (to which the counter overflowed) was actually a legitimate value in the Therac-25 program.

Re:Therac-25 by ars on Tuesday February 18, 2014 @01:18PM (#7259869)
Any chance of an example from a slightly more recent language?

If you are going to say we shouldn't use C because of overflow handling, at least suggest a language that actually does it. Otherwise you haven't really made your case.

Re:Therac-25 by pjmlp on Tuesday February 18, 2014 @02:12PM (#7260321)
Ada is a recent language, just last year got a new standard revision.

Chances are, when you travel by train or take a plane, the control system was developed in Ada.

When human lives are at risk, C is becoming less and less an option.

And when the company still goes C, many countries have certifications in place like MISRA, that make C look like Ada with C syntax.

Re:Therac-25 by ArkyBeagle on Wednesday February 19, 2014 @02:41AM (#7262350)
Language choice is secondary to methodology, but Ada does a darn fine job of checking an awful lot of methodologically useful things up front.

It's less the language than the ecosystem and the infrastructure. It's eminently possible to develop 'C' that's exhaustively tested - I've done it. I am not sure that is true of C++ or Java.

Re:Therac-25 by ars on Tuesday February 18, 2014 @03:36PM (#7260859)
Fine, I'll grant you Ada, and it does seem like a good language for these kinds of things. But you have to acknowledge it's not going to become a popular language for regular software projects.
Re:Therac-25 by pjmlp on Tuesday February 18, 2014 @03:59PM (#7260997)
Sure I won't see the a SV startup using it.
Re:Therac-25 by ars on Tuesday February 18, 2014 @05:26AM (#7257371)
> and moving away from languages like C

If you look at the reports of serious failures like this once (and other famous ones), none of them are related to the use of C.