News for hackers. Stuff that matters.
Re:F-35 radar system has bug that requires hard reboot in fl... by hackuser on Thursday March 10, 2016 @01:21PM (#11260927)
I read elsewhere, from a reliable source that I can't find right now. that the mean time between reboots on 'mature' fighter planes is 8 hours and the F-35 is at around 4 hours so far with plans to improve it.

It was the first number that shocked me the most.

EDIT: I found it:

http://breakingdefense.com/2016/02/bogdan-predicts-f-35s-for...

The toughest problem the program is having is matching the timing of the aircraft’s fusion software with its sensors’ software. “As we add different radar modes and as we add different and capabilities to the DAS system and to the EOTS system, the timing is misaligned,” and then you have to reboot it. Bogdan said he’s aiming for eight to nine hours between such software failures when a radar or DAS or EOTS needs to be rebooted, which is what legacy aircraft boast. Right now they are at four to five hours between such events. “That’s not a good metric.”

I don't understand a lot of that, but I think the fusion softwre refers to a unified UI for the pilot that merges information from the plane's very many sensors and presents it in a comprehensible way on one screen. Apparently pilots in older planes suffer from data overload and from watching many different screens at once - all while trying to fly a plane in combat. UI is reported to be a highly consequential improvement for the new plane.

Re:F-35 radar system has bug that requires hard reboot in fl... by mbell on Thursday March 10, 2016 @02:16PM (#11261417)
> The toughest problem the program is having is matching the timing of the aircraft’s fusion software with its sensors’ software

I don't know if I'm interpreting that correctly but what I'm getting out of it is clock skew between sensors and the fusion system, which I assume to be referring to sensor fusion.

That makes sense, analogous to a typical distributed systems issue. I would imagine the skew needed for the type of sensing they do is quite small. I doubt they can just toss an atomic clock on the plane :)

Re:F-35 radar system has bug that requires hard reboot in fl... by gherkin0 on Thursday March 10, 2016 @02:22PM (#11261460)
> I doubt they can just toss an atomic clock on the plane :)

They totally could. Some atomic clocks are quite small. The one pictured on the below wiki page doesn't look that much bigger than a 3.5" hard drive.

https://en.wikipedia.org/wiki/Rubidium_standard

Re:F-35 radar system has bug that requires hard reboot in fl... by mbell on Thursday March 10, 2016 @03:49PM (#11262170)
> They totally could. Some atomic clocks are quite small.

Are they mil spec? In other words do they operate correctly over a very large temperature range, with large input voltage variance, under high G-forces, in high vibration environments, etc?

I don't know anything about atomic clocks past what I can read on wikipedia but, I can say there is generally a large difference between mil spec ICs and commercial or industrial grade ICs, I would assume this would extend to atomic clock scale devices as well, but just a guess.

Re:F-35 radar system has bug that requires hard reboot in fl... by engi_nerd on Thursday March 10, 2016 @03:58PM (#11262242)
Who needs atomic clocks on individual planes when the US government has spent billions of dollars to place a fleet of atomic clocks in orbit?
Re:F-35 radar system has bug that requires hard reboot in fl... by mbell on Thursday March 10, 2016 @04:03PM (#11262280)
Someone whom wants their planes to work when the enemy jams GPS and/or shoots the satellites down. We're talking about war here after all.
Re:F-35 radar system has bug that requires hard reboot in fl... by engi_nerd on Thursday March 10, 2016 @04:15PM (#11262391)
So you freewheel once GPS goes down. You might drift a few microseconds. This is not necessarily a problem.
Re:F-35 radar system has bug that requires hard reboot in fl... by NeutronBoy on Thursday March 10, 2016 @04:52PM (#11262710)
If you're traveling at high speeds and relying on this timing to adjust flight control surfaces, or trying to use radar to get position of combatants and/or the ground, a few microseconds clock skew would be a major issue.
Re:F-35 radar system has bug that requires hard reboot in fl... by engi_nerd on Thursday March 10, 2016 @06:48PM (#11263516)
I didn't say clock skew. I said drift. As in, you lose GPS timing signals so your own internal master clock freewheels. That has nothing to do with clock skew (different components within the jet 'seeing' different time due to delays or other issues). The jets are designed to function with degraded or no GPS functionality. But, if it's there, they will take full advantage of it.
Re:F-35 radar system has bug that requires hard reboot in fl... by engi_nerd on Friday March 11, 2016 @07:43AM (#11266251)
For those downvoting, see my comment below, where I explain why this really isn't a problem.
Re:F-35 radar system has bug that requires hard reboot in fl... by gozur88 on Thursday March 10, 2016 @06:03PM (#11263229)
Also, if I'm understanding the link properly you'd need an atomic clock for every sensor to make it work.
Re:F-35 radar system has bug that requires hard reboot in fl... by scld on Thursday March 10, 2016 @03:27PM (#11262008)
I wonder how something like that stands up to a gunfire vibration test.
Re:F-35 radar system has bug that requires hard reboot in fl... by mentat on Thursday March 10, 2016 @03:38PM (#11262105)
That might have detection implications though.
Re:F-35 radar system has bug that requires hard reboot in fl... by Johnny555 on Thursday March 10, 2016 @02:21PM (#11261454)
I would imagine the skew needed for the type of sensing they do is quite small. I doubt they can just toss an atomic clock on the plane

I don't think a single atomic clock would do it -- sounds like they'd need a high accuracy clock for each separate subsystem.

But unlike distributed systems, it seems like an aircraft is small enough to have one clock that everyone uses - any idea why they don't do that? Single (or too much of a) point of failure? Disparate vendors can't agree on a standard? Too much extra wiring or other complexity?

Re:F-35 radar system has bug that requires hard reboot in fl... by vonmoltke on Thursday March 10, 2016 @02:30PM (#11261541)
The standard way to do timing in a system like this is for each subsystem to run its own binary clock, with the main system sending period time-of-day messages to keep everything externally synchronized. In that sense, all the systems do have one clock. Many do not need to worry about wall clock time or synchronization, though.
Re:F-35 radar system has bug that requires hard reboot in fl... by luch on Thursday March 10, 2016 @02:55PM (#11261786)
That is one explanation. I've worked in avionics telemetry & remote sensor fusion and there is actually a push to go from synchronous archs (using PCM on I2C buses) towards modular and networked archs (using some variant of IP/etc. protocols).

Problem is you need a reference time clock. On monolithic arch, is easy : the backbone bus timestamp the paquets it sees. On networked arch you need a time server a la NTP, otherwise you have time skew and loopbacks.

Even with a accurate time clock, you may also need to correct for timing issues due to cable length.

Re:F-35 radar system has bug that requires hard reboot in fl... by mbell on Thursday March 10, 2016 @08:35PM (#11264056)
I'm curious what you mean by 'synchronous' here, for example an I2C bus is clocked by the master, ignoring clock stretching and that clock need not be in any way synchronous with the processor clock on the slave.

And to clarify, we're talking about PHY level protocol clocks here, not time clocks.

Re:F-35 radar system has bug that requires hard reboot in fl... by jasonwatkinspdx on Thursday March 10, 2016 @03:27PM (#11262006)
Check out PTP, and then the WhiteRabbit research project. It's not exactly easy to implement, but you can get down to ~10ns sync on an ethernet network using it.
Re:F-35 radar system has bug that requires hard reboot in fl... by CamperBob2 on Thursday March 10, 2016 @03:19PM (#11261967)
I doubt they can just toss an atomic clock on the plane :)

As I understand it, an early variant of the TCAS system for collision avoidance did just that. Every aircraft would have a Cs or Rb standard that was synchronized to a master clock, and they'd broadcast their position and altitude at an assigned time slot.

This would have been very expensive at the time (late 60s-early 70s?) they were proposing it, and it would only have worked if everyone had the necessary equipment. So it didn't fly. But in later decades it was no big deal to put a high-quality atomic time/frequency standard onboard an aircraft, and I wouldn't be surprised if most large airplanes actually do carry a Rb standard for one reason or another.

Edit: the proposed system was called EROS: http://www.allstar.fiu.edu/aero/tcas.htm

Re:F-35 radar system has bug that requires hard reboot in fl... by theoh on Thursday March 10, 2016 @06:37PM (#11263447)
I think the image a lot of us have is that an atomic clock is a PC-to-dishwasher-sized box. Rubidium standards are smaller than that but less accurate than the lab grade equipment.
Re:F-35 radar system has bug that requires hard reboot in fl... by engi_nerd on Thursday March 10, 2016 @03:30PM (#11262032)
You don't really need to toss an atomic clock on the plane. Just make the plane use GPS.

Where one of the tricky parts comes in, is in distributing this clock signal to all portions of the avionics system that need it. But that's beside the point.

It doesn't really sound like a timing skew problem (as in, a problem with making all systems recognize the same time simultaneously) but rather with making sure that all relevant information needed to construct a coherent 'picture' from all sensors reaches the core processors in a timely manner.

You can't provide a real time display if, say, your radar data is delayed for a significant period of time due to problems or instabilities with the radar's onboard processor.

Re:F-35 radar system has bug that requires hard reboot in fl... by w4 on Thursday March 10, 2016 @05:39PM (#11263092)
> You don't really need to toss an atomic clock on the plane. Just make the plane use GPS.

These are warplanes - you don't want critical systems reliant on satellites that could be jammed, or downed, during wartime, which I suspect is why they don't do this already.

Re:F-35 radar system has bug that requires hard reboot in fl... by engi_nerd on Thursday March 10, 2016 @06:50PM (#11263522)
But fighters do use GPS. Daily. They use it for accurate navigation. But they do not absolutely depend on it.
Re:F-35 radar system has bug that requires hard reboot in fl... by gherkin0 on Thursday March 10, 2016 @09:13PM (#11264208)
> But fighters do use GPS. Daily. They use it for accurate navigation. But they do not absolutely depend on it.

But...

>>> Just make the plane use GPS.

...implies you would make them absolutely depend on it. Military aircraft use GPS daily, because it's cheaper and better. However, they also have accurate inertial navigation systems as a backup, so they can still operate effectively if the enemy makes GPS unavailable.

Re:F-35 radar system has bug that requires hard reboot in fl... by jacquesm on Thursday March 10, 2016 @01:39PM (#11261082)
How many fighter plane flights last longer than 8 hours? It's a mean time after all, it could simply be a reflection of the length of the sortie.
Re:F-35 radar system has bug that requires hard reboot in fl... by bluedino on Thursday March 10, 2016 @01:59PM (#11261276)
Probably not a lot but it wouldn't be uncommon for an American plane to be in the air for 12+ hours with in-air refueling.
Re:F-35 radar system has bug that requires hard reboot in fl... by neurotech1 on Thursday March 10, 2016 @02:36PM (#11261618)
It is relatively uncommon for a single-seat (& single-pilot) fighter jet to fly more than 8-9 hours. If they do, then AFI & OPNAV duty rules restrict how these flights are conducted (including combat ops) and how soon the crew can fly their next mission. Aircraft like the B-2 have two pilots and planned sleep periods.

According to references, 13.5 hours [0] is the record for the F-16 combat mission, and this was actually a SEAD (Suppression of Enemy Air Defenses) mission in a single-seat F-16CJ. Similar missions in the EA-18G extended to 13+ hours with a single pilot, along with a WSO/EWO in the back seat.

Various unofficial reports indicate F-16s do fly 14 hour ferry flights, but that is highly unusual.

[0] http://www.f-16.net/f-16-news-article4413.html

Re:F-35 radar system has bug that requires hard reboot in fl... by Grishnakh on Thursday March 10, 2016 @03:15PM (#11261937)
What I'm wondering is how these fighter jet pilots deal with needing to go to the bathroom. 14 hours stuck in a tiny cockpit kinda makes it hard to relieve yourself...
Re:F-35 radar system has bug that requires hard reboot in fl... by cvwright on Thursday March 10, 2016 @03:47PM (#11262160)
Not sure why you are being downvoted. It is a serious question, if a bit gross.

At the Evergreen Air & Space museum (home of the Spruce Goose), they have an old SR-71 on display. Part of the exhibit has a TV playing a video loop with pilots and crew talking about their experiences. One of the pilots recalls how they'd have a "high-protein, low-residue" breakfast like steak & eggs before a long mission. The idea was to provide long lasting energy without having to go "number two" in the suit. He says that some of the crew who would help them out of the flight suit at the end of a mission were female, so it'd be especially embarrassing if you'd pooped yourself. I got the impression that, while rare, it did happen sometimes. :-\

Re:F-35 radar system has bug that requires hard reboot in fl... by mfoy_ on Thursday March 10, 2016 @02:16PM (#11261414)
Forget fuel, and radar crashes... can you imaging flying a fighter jet for that long? The stress on the pilot must be insane.
Re:F-35 radar system has bug that requires hard reboot in fl... by kifler on Thursday March 10, 2016 @02:36PM (#11261620)
Thats why they're fed a cocktail of amphetamines pre-flight for longer sorties.
Re:F-35 radar system has bug that requires hard reboot in fl... by jacquesm on Thursday March 10, 2016 @02:09PM (#11261354)
That mean time still leaves room for that doesn't it?

(2 + 2 + 2 + 2 + 12) / 5 = 4.

And so on. This whole statement is meaningless without more information about the distribution of lengths of flights.

Re:F-35 radar system has bug that requires hard reboot in fl... by aries1980 on Saturday March 12, 2016 @06:02AM (#11272337)
Imagine the Falkland War's Vulcan march with hard reboots...
Re:F-35 radar system has bug that requires hard reboot in fl... by thearn4 on Thursday March 10, 2016 @02:15PM (#11261405)
This is what happens when you have an entire industry (defense/general gov't services) that treats software development as a cost center instead of a profit center.
Re:F-35 radar system has bug that requires hard reboot in fl... by wehadfun on Thursday March 10, 2016 @03:47PM (#11262153)
I think it is time for the incumbents to be interrupted. Boring/Northrowup/... are fking up to much. Here is another defense industry f-up [0]

[0] http://abcnews.go.com/US/jet-skier-broke-jfk-airports-securi...

Re:F-35 radar system has bug that requires hard reboot in fl... by agentx3r on Thursday March 10, 2016 @02:11PM (#11261373)
Likely refers to sensor fusion (https://en.wikipedia.org/wiki/Sensor_fusion), which is an algorithmic process.

A 4-hour MTBF window (even for a subsystem) is terrifying considering modern vehicles like this are likely all fly-by-wire.

Re:F-35 radar system has bug that requires hard reboot in fl... by toomuchtodo on Thursday March 10, 2016 @02:14PM (#11261398)
Don't think of it as a failure. Think of it as a garbage collection pause at subsonic speeds.
Re:F-35 radar system has bug that requires hard reboot in fl... by frankydp on Thursday March 10, 2016 @02:26PM (#11261510)
Spot on.

This article should clarify what the radar reset that they are referring to is. Integer, origin, and timing resets are a function of radar tech, and are not specifically related to something like a system reboot. This would be something like restarting a process due to bad data getting into your process from a sensor and potentially leading to a bad result from a range or trajectory.

Re:F-35 radar system has bug that requires hard reboot in fl... by hackuser on Thursday March 10, 2016 @02:14PM (#11261394)
From the quote it appears that it's 4 hours between reboots of any subsystem, not the whole plane.
Re:F-35 radar system has bug that requires hard reboot in fl... by nerdy on Thursday March 10, 2016 @01:40PM (#11261097)
I have no experience with aviation, why do they need to be rebooted at all in less time than they can fly without refueling? It would seem with such a critical system the tolerance for rebooting would be exceptionally low. The F35 is not a dogfighter; does it have any meaningful capabilities while radar-blind?
Re:F-35 radar system has bug that requires hard reboot in fl... by artmageddon on Thursday March 10, 2016 @02:16PM (#11261412)
> why do they need to be rebooted at all in less time than they can fly without refueling?

I don't know much about Navy / Air Force ops, but it's possible that a sortie may consist of taking off and returning to rearm / refuel without shutting down, which could easily span more than four hours. In addition, lots of fighter planes do refuel in mid-air, so the requirement is still very prevalent for the subsystem to remain functional.

Re:F-35 radar system has bug that requires hard reboot in fl... by neurotech1 on Thursday March 10, 2016 @03:15PM (#11261932)
Most military jets can be hot refueled on the ground, with engine(s) running. Some fighter jets (F/A-18s etc) can hot swap crew on the ground, with one engine running.

With the F-35 and other jets, its possible to swap crew and perform minor maintenance with the jet electrically powered up and cooled with auxiliary air supply, although the engine is shutdown and then restarted. This actually saves significant time on the pre-takeoff procedures. I would think the Lockheed Martin engineers could also perform diagnostics after a failure, without interrupting power or restarting the sub-system.

Re:F-35 radar system has bug that requires hard reboot in fl... by nerdy on Thursday March 10, 2016 @02:33PM (#11261569)
I'm actually curious what possible technical reason would prohibit them from running for 48+ hours (or indefinitely). IE some kind of garbage collection issue. I'm unaware of any reason they should ever have to be rebooted aside from software upgrades and curious why it might be necessary (technical reason). Unfortunately, from the article: "But the Air Force did not go into detail about when the instability in the radar system occurred..."

Edit: It is mentioned elsewhere in this thread that it is a timing issue, see: http://breakingdefense.com/2016/02/bogdan-predicts-f-35s-for...

Re:F-35 radar system has bug that requires hard reboot in fl... by vonmoltke on Thursday March 10, 2016 @02:46PM (#11261700)
Have to shut the engines off at some point. Any vehicle "reboots" when you shut down the power.
Re:F-35 radar system has bug that requires hard reboot in fl... by vonmoltke on Thursday March 10, 2016 @02:47PM (#11261714)
It is certainly possible, but not normal. Normally the engines are shut off during servicing and the aircraft is hooked up to a ground cart to provide limited systems functionality.
Re:F-35 radar system has bug that requires hard reboot in fl... by saalweachter on Thursday March 10, 2016 @04:14PM (#11262385)
The thing I'm struggling with is, "Why does rebooting solve the problem?"
Re:F-35 radar system has bug that requires hard reboot in fl... by raverbashing on Thursday March 10, 2016 @02:23PM (#11261469)
I can only imagine such sad state of affairs comes from extremely quirky requirements and very tight and crummy process (from TPS reports to worse)

Yes, I can understand the problem is complicated, but come on.

Re:F-35 radar system has bug that requires hard reboot in fl... by dokument on Thursday March 10, 2016 @01:40PM (#11261099)
Any clue to why it would take so long to reboot?
Re:F-35 radar system has bug that requires hard reboot in fl... by shifter on Thursday March 10, 2016 @01:42PM (#11261110)
The reboot itself is not the N hours figure, that's the interval between reboots.