News for hackers. Stuff that matters.
Re:The Patriot Missile Failure by danbruc on Monday March 24, 2014 @12:46PM (#7459276)
This makes no sense in the way it is stated - the calculations involved are independent of the up-time. You do not aim at a target differently depending on how long the system is up. The correct way to think about this is probably the following - they noticed that the up-time drifted and tried to improve on that, but they failed to do so in all places. In consequence different parts of the system used different times that also advanced with differing speeds and this inconsistencies caused the calculations to go wrong. As mentioned in the article, would they not have made the "improvement" all parts would have used the same time and the errors would have canceled because the calculations are time-independent and the drift is probably to small to be important during the relatively short time a target approaches. This seems one of the rare cases where blaming math with limited precision is wrong.
Re:The Patriot Missile Failure by jessaustin on Monday March 24, 2014 @01:14PM (#7459510)
Do you have additional information about this incident? Although TFA alludes to something like what you're saying with its "inaccuracies did not cancel" comment, your point seems to require an assumption that is not stated there. Aren't you assuming a non-distributed system, when it seems very likely that radar sensors and missile launchers would not be co-located? In a distributed system, we certainly cannot assume identical boot dates, and so an accumulating error would accumulate differently at different components of the system.

Systems like this must have time agreement among their different components. It seems that a system developed later would have just used GPS to prevent this problem, but I wonder why they didn't use NTP?

Re:The Patriot Missile Failure by danbruc on Monday March 24, 2014 @01:49PM (#7459875)
The obvious evidence are the laws of physics which are invariant under time translation. What you are saying is exactly what I said - it does not matter if the drifting clocks are in different parts of a distribute system or if a single system uses different clocks. In the end the system fails because different clocks disagree on the current time, not because they drift away from the actual time. And the article states exactly this - they tried to fix the clock drift relative to the actual time but failed to do so in all components and thereby inadvertently introduced two clocks drifting relative to each other.
Re:The Patriot Missile Failure by jessaustin on Monday March 24, 2014 @03:11PM (#7460611)
In the end the system fails because different clocks disagree on the current time, not because they drift away from the actual time.

The possibility I mentioned was that the clocks disagreed because they had drifted away from actual time at similar rates, for different periods. [r ≠ 1 ∧ t₀ ≠ t₁] → [t₀ + (t-t₀)r ≠ t₁ + (t-t₁)r]. Unless specifically addressed, this phenomenon will occur in distributed systems.

That said, the link 'gibrown provided seems to establish firmly that this particular error resulted from times being up-converted (24bit fixed to 48bit floating) via different methods in different parts of the system.

Re:The Patriot Missile Failure by danbruc on Monday March 24, 2014 @04:32PM (#7461233)
Now I see what you mean, equal drift rates but different up-time. Assuming two clocks coming up at different points in time this will indeed cause time differences if they are initially synchronized with a third clock. If the clock coming up later synchronizes with the other clock already running there should be no issues though.

I read the article, too, and this two different ways of converting are more or less equivalent of two clocks starting in sync but drifting relative to each other. Therefore my initial comment was quite on the spot.

Re:The Patriot Missile Failure by Varcht on Monday March 24, 2014 @03:09PM (#7460594)
Yep, that's how I read the article, the target path prediction system was 0.34 seconds out of sync with the targeting system due to differing implementations.
Re:The Patriot Missile Failure by msmillie on Monday March 24, 2014 @01:38PM (#7459749)
The following scenario helps explain: you have two radars. One is a wide-angle, general radar, which sees a missle at some coordinates, travelling at some speed.

The other radar is for target acquisition, has a very narrow spread ("range gate"), and must determine the precise location of the missle given an initial set of coordinates, a velocity, and a timestamp.

In other words, the system is actually aiming at targets differently depending on how long it has been up, tracking so far in front (or behind, I don't recall the details) that it can't acquire the target.

Re:The Patriot Missile Failure by danbruc on Monday March 24, 2014 @01:55PM (#7459934)
You are describing a problem where two clocks drift relative to each other, not a problem where one clock drifts away from the actual time. On the other hand the article gives the impression the failure occurred because the system failed to exactly measure the up-time, not because clocks in different systems or system components drifted relative to each other.
Re:The Patriot Missile Failure by vonmoltke on Monday March 24, 2014 @02:57PM (#7460485)
That is not the impression I got from the article. No logic in the system cares how long it has been up, not directly at least. What matters is drift from its time reference, which is a function of uptime.

Various modules in a complex system like this each have their own clock, which I will refer to generically as a real-time binary counter (RTBC), which the module uses as its event time reference. The RTBC starts at 0 when the module comes up. At some point shortly after coming up the module will check in with its controller, which will send a time-of-day (TOD) message. The module links the TOD message to a particular RTBC tick to create its time reference. At this point the time is free to start drifting relative to the actual wall clock time, until the system is power cycled again.

Re:The Patriot Missile Failure by danbruc on Monday March 24, 2014 @03:19PM (#7460691)
That is exactly what I said - different clocks drifting relative to each other. It is completely irrelevant that their one tenth of a second was not exactly one tenth of a second, what matters is that different clocks in the system had different ideas of one tenth of a second.
Re:The Patriot Missile Failure by darkmighty on Monday March 24, 2014 @05:20PM (#7461588)
You're applying a principle too broadly. Although the laws of physics don't change under a linear expansion of time, they are, for example, sensitive to linear expansion of velocities of missiles only: any non time-linear effect on the velocity is going to impact you -- for example, reynolds numbers for air depend non-linearly on the velocity which may be varying with time. Sure, if you multiply the whole system you would have that compensated by the increase in temperature and pressure which a faster time reference would observe, but it's not simulating the universe, just a limited set of variables.

Also, for obvious reasons of consistency and precision it would be better to keep a standard reference regardless.

Re:The Patriot Missile Failure by danbruc on Monday March 24, 2014 @09:42PM (#7463185)
I do not think of the problem as scaling the time by a factor - although this is the correct description - but as adding a constant offset. I think this is justified because the small drift is not significant during the relative brief period of time a target approaches. The offset builds up over time but only in the parts of the system that did not receive the improved algorithm and therefore these different parts disagree more and more on what the current time is.
Re:The Patriot Missile Failure by vonmoltke on Monday March 24, 2014 @04:15PM (#7461099)
> It is completely irrelevant that their one tenth of a second was not exactly one tenth of a second.

Its very relevant when the module that is off is trying to make telemetry calculations based on target Doppler velocity, which is given with real, ISO standard seconds. There is no clock involved in that. Diverging module clocks amplifies the problem.

Also, the ultimate reference is the true definition of a second. All modules are expected to be using it, as it is used to synchronize modules. It is the clock and at some level a clock that has a faulty definition will be drifting off another clock. Your distinction is irrelevant as far as real-time systems are concerned.

Re:The Patriot Missile Failure by danbruc on Monday March 24, 2014 @04:53PM (#7461377)
You are making a lot of assumption about how time might be used, nut I will ignore that because I have no clue if that is what really happens.

Let me repeat my point clearly. All clocks will drift away from the actual time. All the physics involved and measurements done are not depended on the current time - they will work the same at 14:07 as they do at 23:51 and they will therefore also work the same when the clock of the system drifted away from the actual time and believes it is 12:34 while it is 12:35. Important is only that all parts of the system agree on what the current time is and that the clock does not drift at such an high rate that all measurements and calculations done during a brief period of time become invalid, i.e. the clock should not report that it took two seconds for the incoming missile to travel one kilometer while it took only one second.

And the article gave the impression - at least to me - that the failure was caused because the system believed to be up for 100 hours while it was up for 100 hours and 340 milliseconds longer due to an imperfect representation of one tenth of a second. This makes no sense and is not what caused the failure. The failure was caused - as detailed in the other linked article - because one part of the system believed to be up for 100 hours while another part of the system performed more precise time conversions and knew that it was up for 100 hours and 340 milliseconds and this time difference between two parts of the system caused the failure.

For example one part of the system may have decided that the missile should be launched at 12:00:00.000 and the system responsible for doing so did that according to its clock but because of the time difference it was at 12:00:00.340 according to the clock of the system that made the decision.

Re:The Patriot Missile Failure by krfsm on Tuesday March 25, 2014 @09:17AM (#7465613)
My interpretation of the article:

Time is kept as an integer, stepped ten times per second. This can be exactly represented as a float, so probably uses the same 24 bit register. For 100 hours this integer would be 3600000, which fits into 24 bits with some room to spare. (But it would give a max uptime of the system of about 466 hours.)

Wide arc radar notes location, velocity, and time from clock above. This output data is still good enough for pinpointing the next position with a precision of about 170 meters (the distance the scud travels in the 0.1 second step of the clock). The precision radar system probably had accounted for this, and had a wide enough beam to handle this case.

Now, when deciding where to point the next precision beam, the radar multiplies the stored time value (exactly 3600000) with 0.1 (which is not represented exactly, but instead is about 0.000000095 less than 0.1) and uses this computed value in further calculations. This floating point value is now 0.34 seconds less than expected. The precision radar, even though it uses the same clock as above, has an incorrect representation of when the last wide arc radar update took place, and this propagates to the prediction of where the scud will be next (which is now off by 0.34 * 1676 ~= 570 meters). Thus, when it points the beam to where it believes the scud will be, the scud is outside the precision radars cone.

Note that both wide arc and precision beam systems have exact knowledge of the current system time at the point of their respective operations. What fails is precision beam's calculation of what wide arc's time reference actually meant.

The ironical part in the article probably refers to some computation using a delta, and if both time references ("then" and "now") have the error the delta will be small and possibly insignificant. However, if "now" is replaced with a more accurate representation of the clock above, only "then" has the big error, and the delta will be just as far off as the incorrect value above.

The exact error propagation depends on the order the calculations are performed in, and there's a whole field (numerical analysis) dedicated to controlling these errors. As developers we gladly ignore the problem even when we shouldn't.

Re:The Patriot Missile Failure by grkvlt on Monday March 24, 2014 @05:56PM (#7461916)
> You are describing a problem where two clocks drift relative to each other, not a problem where one clock drifts away from the actual time

The latter is simply a specific case of the former. The second clock is the one measuring 'actual' time, and the drift is relative to it.

Re:The Patriot Missile Failure by danbruc on Monday March 24, 2014 @09:47PM (#7463217)
Of course, but the point was, that a drift relative to actual time does not cause problems as long as all clocks in the system drift at the same rate while the failure was caused by different clocks in the system drifting relative to each other and therefore with different rates relative to actual time. Therefore I treated actual time as a special clock.
Re:The Patriot Missile Failure by gibrown on Monday March 24, 2014 @01:55PM (#7459936)
A better description of the bug: http://www.ual.es/~plopez/docencia/itis/patriot.htm

It was related to limited precision, but was also related to using different calculation methods in different parts of the code.

Re:The Patriot Missile Failure by fennecfoxen on Monday March 24, 2014 @12:56PM (#7459358)
> This makes no sense in the way it is stated - the calculations involved are independent of the up-time. You do not aim at a target differently depending on how long the system is up.

At some point or another your anti-SCUD computer is going to have to say "Send the electrical signal to launch this missile NOW!". I assume this is where they make use of the system clock.

Re:The Patriot Missile Failure by danbruc on Monday March 24, 2014 @12:59PM (#7459389)
Definitely, but it does not matter what the value of NOW is as long as all parts use the same time. Only their attempt to fix the drift caused different parts to have a different understanding what NOW = 5484515 means and that caused the problem.
Re:The Patriot Missile Failure by salem on Monday March 24, 2014 @01:48PM (#7459867)
My understanding is that they used a floating point value for time measurements, and they used time since boot as a reference. After a long enough time since boot, there were not enough significant digits available to represent both time since boot, and accurately capture the different in time between two events.

The first Ariane 5 launch also had a failure relating to floating point math, in that case an overflow when converting to an int16:

http://www.di.unito.it/~damiani/ariane5rep.html

Re:The Patriot Missile Failure by danbruc on Monday March 24, 2014 @01:50PM (#7459891)
The article states they use a 24 bit fixed point format.
Re:The Patriot Missile Failure by salem on Monday March 24, 2014 @06:24PM (#7462110)
Sure, but the point is that it did not have enough precision for both a large absolute value, and small time measurements, when based on time since boot.
Re:The Patriot Missile Failure by danbruc on Monday March 24, 2014 @09:33PM (#7463150)
No, this is not what happened. They converted the 24 bit fixed point value into a different format using different algorithms in different places of the system resulting in slightly different values. The failure was then caused by the difference between these values, i.e. different parts of the system did not agree on what the current time is.