When I worked on military autonomous vehicles (which I’d expect to be similar to these) we always had at least two system on board: 1) A real-time flight controller, 2) One or more Linux computers, networked via an on-board LAN which performed all the other tasks.

I worked with some people of that particular JPL software team to setup a similar setup like the Mars helicopter for a earth UAV.

Can confirm that multiple real time systems are used. They are controlled by a non real time Linux system.

VxWorks has been battletested in NASA space probes for over 30 years.

https://en.wikipedia.org/wiki/Comparison_of_embedded_compute...

I recall an early bug in the 2004 MERS duo. They were the first to use flash memory and its new drivers. The file freelist busted and the OS kept on rebooting. Fortunately a fix was uploaded from Earth.

https://en.wikipedia.org/wiki/Spirit_(rover)#Sol_17_flash_me...

Fun read, tyvm! I loved this anecdote from the second link:

> On sol 20, the command team sent it the command SHUTDWN_DMT_TIL ("Shutdown Dammit Until") to try to cause it to suspend itself until a given time.

>

> It seemingly ignored the command.

The fix was basically, there are too many files in the flash drive, causing it to reboot endlessly because the booting process could not handle so many files in the flash. Solution was to delete some files.

Mo' files, Mo' problems.

the pathfinder also had a priority inversion bug resulting in deadline trips. same story, they patched it...

Real-time Linux is a thing, although it was maintained separately, outside mainline Linux, for a long time. About 4-ish years ago, the project got some decent funding and is now part of Linux Foundation:

https://www.linuxfoundation.org/en/blog/real-time-linux-cont...

I worked at NASA Armstrong for a summer. The have an entire team there that at least a few years ago worked full time on real time Linux. IDK if that was used in this case or not.

The helicopter uses a Qualcomm Snapdragon 801, which doesn't appear to be radiation hardened. I'm guessing their usual BAE RAD750 PowerPC rad hardened CPU was too heavy to put on the helicopter.

another thread noted that the hardened CPU was not fast enough to do the required sensor fusion and responses. In the case of a severe fault, the 801 can be restarted fast enough (while in mid flight!) to not crash the vehicle.

I think the primary reason is that Ingenuity is not considered essential part of the main mission, so they could use non-qualified COTS parts like Snapdragon and Linux.

The Helicopter is also not operated in space, so radiation might be less of an issue. It's intended only as a flight demo anyway.

The Martian atmosphere and magnetosphere are both substantially worse at blocking radiation than those of Earth, so I'd imagine radiation would, at best, only be marginally less of an issue than, say, in interplanetary space.

If memory serves, the radiation environment of the Martian surface is not too much worse than LEO. If so, radiation would probably not be that much an issue.

More likely just too power hungry

apparently list price for the RAD750 is $200,000! could you not externally harden the snapdragon for less?

That's not how hardening chips works.

You need a specialized fab manufacturing, etching and packaging process for this and it's not like regular fabs are cheap to begin with. Plus you're working from the start with much larger nodes with dedicated cell libraries so everything has to be designed from scratch to fit that node which means you can't reuse consumer off the shelf designs very easily.

Then there's the lack of economies of scale in building such custom parts in small numbers. I imagine if Apple would only order 100 5nm chips per year from TSMC, the unit price would be equally eye watering.

I've only read this doc quickly [1]. I would have guessed a lead case of a particular thickness would do the trick. But I don't know the thickness needed, maybe too much?

[1] https://nepp.nasa.gov/files/25295/MRS04_LaBel.pdf

In theory you could just cover everything in lead and call it a day, IF, lead wouldn't also be one of the heaviest metals in the world, which kinda goes against the mantra of space travel.

Actually it doesn’t work that way (solely based on mass). Radiation doesn’t just get stopped by lead, it generates secondary particles. Lead happens to be effective against x-rays, which is why we tend to think of “lead shielding” but it’s not effective against all types of radiation. So it would help but you couldn’t just call it a day, and other materials are more mass efficient.

yeah, but it'd only be 3/8 as heavy on Mars!

seriously, that's something I haven't thought much about. when designing rovers, do they calculate solely on the weight of what it will be on the destination, or limit it to weight limits of escaping earth's gravity well?

Rover designer here. The weight at the destination is important to characterize performance but it’s not a big driver of design. For example, once you consider losses due to friction inside gearboxes, there’s no big difference to size a motor. You want to be able to test on Earth easily anyway, so you wouldn’t design without a margin to allow that. I suppose if you designed a rover for Uranus you might take a different approach.

Launch mass is important as a constraint, but the launch environment is the main design driver for mass. The rover must be much stiffer than the rocket to not “couple” it’s response. The first vibration mode (think tuning fork) of a rocket is about 20Hz so the spacecraft inside needs a first vibration mode higher than 40Hz. Something inside the spacecraft similarly needs a first vibration mode higher than 60Hz or 80Hz, although you can make exceptions based on analysis.

But that’s not all, the sustained load on components during launch can easily be 10 to 30 times gravity (30G) and the instantaneous load can be 100G. You can’t just add material for these kinds of loads, it would be an endless feedback loop because adding mass decreases stiffness. Look closely and you’ll see that every deployable or movable part of the rover is locked down by a mechanism until it has landed.

It's not just the mechanical weight at the destination and the work required to escape earth's gravity (which is enormous). Mass of the rover also means more fuel to speed up and slow down if there is any delta-V changes en-route, the heat energy that needs to be dissipated during re-entry, the forces on parachute and lander as well. So really, that mass penalty gets paid over and over... and not often linearly.

Weight on board the rocket is the most important consideration as payload weigh is the limiting factor.

Lead coating a critical and embedded component like a CPU, or even a SOC, has gotta be barely significant for the weight of the payload.

I'm guessing the faster CPU is just not necessary for the core rover, so via KISS, use the proven chips.

Ha, if you think that's a lot, I've got some news for you. That might be the price for just the CPU, but the price for the avionics package (basically a RAD750 with the necessary boards to manage power and IO) is a whole lot more than that. (I'm probably not allowed to name the exact number, but you're going to need to add a zero for sure.)

no way I'd be surprised by it. When you only sell 4-5 a year, its going to be pricey!

Is it known what flavor of Linux they're running? E.g. are they using real time modifications to the kernel or something?

The description "real-time" varies in meaning depending on the application and acceptable average and maximum latency.

The un-informed usually thinks "real-time" means "hard real-time", but that's seldom necessary, so one can save on additional expense and effort by using a regular linux distro and removing most of the daemons and file location indexing.

I've done real-time development (for Space Shuttle, rocket and balloon projects), and largely all I care about is if a circular buffer can be emptied in time before it fills. That's one technique for avoiding latency variation issues.

The versions of linux that you would normally encounter aim for music real-time, which is about 10 ms latency. 30 ms is considered to be bad.

Most of the pro Yamaha synths use linux as the embedded OS, and some of the code is downloadable (they attempt to comply with the letter of the GPL.) So you can go down to Guitar Center and do a real-time test anytime yourself. :)

The iPhone is pretty good for music, because it was designed to have low latency when playing.

https://www.synthtopia.com/content/2018/02/17/10-years-later...

https://superpowered.com/androidaudiopathlatency

Looks like Wind River discontinued RTLinux, which was hard real-time (a real shame actually, as it removes one of the few hard real-time options):

https://en.wikipedia.org/wiki/RTLinux

As someone that worked at Microsoft on the XAudio 2 for the Xbox 360 & Windows Phone stacks... holy heck that android audio path is painful to read. >.<;;

We went with a 5.33ms quantum on the Xbox 360 (vs. Window's 10ms), and tried to stay out of the way as much as possible to ensure minimal latencies: https://docs.microsoft.com/en-us/windows/win32/xaudio2/xaudi...

The Akai MPC X and its siblings run Linux too. Based on buildroot according to someone that looked into it.

https://niklasnisbeth.gitlab.io/mpc-internals/

However I don’t remember the GPL being mentioned anywhere when I had my MPC X. I sold it recently because of not having any money. But I hope to own one again in the future. If I ever do have one again I will probably have a closer look at what it says about the GPL, and then try and get a copy of all of the open source portions of the firmware directly from Akai.

Just as an aside, you probably (and hopefully) don't need to have the unit to download the manual—which is where the licenses are likely mentioned. Though this one seems to direct the user to Akai's site.

E.g. https://www.mpc-tutor.com/akai-mpc-manuals/

> Looks like Wind River discontinued RTLinux

Xenomai is still under somewhat active development.