This stuff really does get complex. A sensor controller will likely be on multiple cycles internally: one for oversampling the sensor hatdware and one for transmitting the (filtered/corrected/calibrated) results. A "fuser" as you call it (never heard that term before) needs to make sure that it does never act on stale sensor information (sensor malfunction, accumulated communication issues). Transmission errors need to be detected. Random bitflips in values that are stored in volatile memory for long time spans need to be checked and acted upon.
Every independent controller in such a system requires some kind of watchdog that needs to be reset periodically. Too many watchdog resets in a row indicate a failure and the affected system must shut down in a defined way. You need ways to deal with any combinations of controllers going belly up and avoid taking unsafe actions. For many systems transitioning into a totally inert safe mode is sufficient, but not always.
All of the hardware must constantly run self tests. That includes periodic CPU and memory tests (both volatile and non-volatile memory) and also all periphery that us involved. If, for example, a DAC is used to send a signal, the resulting signal must be read back by different hardware to check that the generated voltage is indeed correct.
Manually threading together all these different kinds of cycles and asynchronous events without a RTOS scheduler is hard and becomes error prone. The result is likely less resilient than a preemptively multithreaded firmware.
An operating system, real time or otherwise, handles activating processes, IO, and interprocess messaging. That's it. You don't get magical serial line noise clearing pixies with it, and it doesn't make your actuators less drunk.
And...
> Manually threading together all these different kinds of cycles and asynchronous events without a RTOS scheduler is hard and becomes error prone.
You just said "getting input and sending output is way too hard for software on a Linux kernel." That's a crazy person statement. It turns out that Linux is, and has been for a loooong time, very good at doing operating system things like activating processes and interprocess messaging.
> The result is likely less resilient
Saying "likely" here suggests that you don't know what the result actually is. So what are you arguing? What was my statement?
Whatever assumption you're making that says "This. This right here is the reason why we definitely need an RTOS." Just don't make that assumption. That assumption is wrong.
So to me, "you don't need an RTOS" means that you're running on bare metal. And that would be hard to pull off for the reasons that I outlined above. And I think this is where we ended up misunderstanding each other.
I enjoy the kinds of restricted RTOS environments that we use because their simplicity means that I can get a total understanding of what is going on quite easily.
This does not mean that Linux is completely inappropriate for real time tasks. I am sure that you could analyze and patch the kernel to match pretty high standards (others mentioned patches). Given the relative size and complexity of a Linux system, this is no simple task. But if you run it on appropriate hardware (not your run of the mill x86), I don't see why you couldn't get reliable realtime responses.
But safety essentially means that the software will not fail more often than once every x hours where x ranges between 10^5 and 10^8, depending on the level of safety required. Proving that for a complex system is hard. For example, how do you show that the essentially indeterministic pattern of dynamic memory allocations happening in a Linux system will never lead to memory exhaustion by fragmentation?
I know of no version of the Linux kernel (or GCC, for that matter) that got a functional safety certification. Safety standards are transitioning away from allowing positive track records as sufficient proof that a piece of software meets safety standards. Do-178 now only allows certified software AFAIK and I expect this to be carriedover into ISO 61508 and ISO 26262. This means a regulated development process, pretty strict coding standards, complete test coverage, full documentation, and so on also for all 3rd party software. Not sure how this transition is going to play out in practice.
Do you know any ASIL-C or ASIL-D (or SIL-2/SIL-3) software that is running on Linux? I am curious whether anybody managed to get that certified. I know that Linux is running on some class II medical equipment, but then, standards for these devices are inexplicably lower in practice in my experience.
A stock modern Linux kernel on almost any hardware platform will give millisecond level responses. Much of the old PREEMPT_RT patch set features from the old 2.6.x days for real-time response has been merged to the mainline kernel.
There are lots of problems in software where you are controlling something physical with a control loop from 1 to a couple hundred Hz. Many people assume a hard real-time deadlines are necessary for this sort of system, but through good system design practices it often is not necessary. For example, if something physical must be sampled with very low jitter, let some hardware do the sampling and latch it in a register and then let the software come in with a variance of hundreds of microseconds to get its work done. Once again, write the output to a latched register and let hardware worry about taking the shadow register with very low jitter.
Having worked on bare metal microcontrollers, to various RTOSes, to higher-performance embedded CPUs with Linux, I prefer Linux on higher-performance hardware. Obviously, this isn't always possible, especially in power constrained situations. But with Linux, when you suddenly need to have support for an arbitrary network protocol, a database, a filesystem, graphical output, etc. you can have something together in no time. It is often a monumental effort for such a task when bare metal or with a RTOS. It is often difficult to get the supporting software and libraries to build on an RTOS in the first place.
Here's what's inside of every autonomous vehicle ever made: a message-passing subsystem, sensors, fusers, navigation, dynamic control, actuator device drivers, and thruster device drivers.
Sensors measure things and emit readings. Your most expensive, highest frequency general purpose sensors emit new readings at something too fast for a human but hella slow for a computer, like 100Hz-5KHz. Your common sensors, a video camera for instance, don't get even close to that. These sensors are often connected, even today because milspec companies hate modernity, via RS-232 serial cables. For those younger than 30, RS-232 is what non-Apple computers used for non-keyboard/mouse peripherals prior to the introduction of the first iMac in 1998 because USB didn't really take off until then.
Sensors send their readings via the message-passing subsystem to fusers.
Fusers take the readings from the sensors and, hur hur, "fuse" them together into a description of where the vehicle is and what the environment is like. This usually involves something like a kalman filter. Fusing even your very fastest sensors, the 5KHz IMUs of the world, is just a small bit of math and basically takes no time at all.
Fusers send their fused states via the message-passing subsystem to navigation.
Navigation takes the fused sense of self and the world and decides which direction to head and how fast to go. The objective could be something like hitting route waypoints or it could be something like staying in a lane and not being rear-ended and avoiding obstacles. Car navigation probably doesn't act on new input more frequently than 100Hz, you certainly can't act on new input more frequently than 100Hz, and it takes basically no time at all.
Navigation sends its directives via the message-passing subsystem to dynamic control.
Dynamic control takes navigation's "which way" and "how fast" directives and turns them into more realistic short-term goals accounting for hysteresis and other physical limitations of the system like minimum turn radius. This is just a small bit of math and basically takes no time at all.
Dynamic control sends its directives via the message-passing subsystem to the actuator and thruster drivers.
Actuator drivers convert dynamic control's "go more left" message into trying to go more left.
Thruster drivers convert dynamic control's "go more fast" message into trying to go more fast.
Actuator and thruster drivers send readings (hopefully) from the actuators and thrusters, because those are also sensors, back to dynamic control and fusion.
Sensors feed into fusers, fusers feed into nav, nav feeds into dynamic control, dynamic control feeds into actuation and thrust. When you have new data, you do something new with it which is technically doing the same old thing with it and just producing new output.
Now there aren't that many sensors. There are way fewer fusers. There's only one navigation. There's probably only one dynamic control, though there could be a couple.
Anything else that I haven't already described, like Waymo's machine learning object classifying 4D mustache adding hotdog detectors, are just sensors and fusers sitting on their own computers feeding new lat/lng/heading/speed to navigation at a rate that is hella slow for a computer. And for sure Waymo's convolutional neural network middle-out jaywalking yoga mom detector takes a lot of processing, but it's running on its own computer, not competing for resources, and emitting its fused readings at some hella slow for a computer rate.