rt linux is not appropriate for many hard realtime requirements

i have seen some try and fail in an effort to save $perceived_value

Is there a good libre real-time OS you would recommend over it?

RTEMS is one and there is also eCOS. RTEMS has flown in space a few times so it is proven.

Forgot to mention: the neat part of RTEMS is it allows one to sort of turn a Unix program into a bare metal program.

Why do RTEMS and eCOS trump, say, Linux? Linux on a RPi has also famously flown (really!) in space.

It's all very well asserting something but a bit more chat would be nice.

I'm not sure what "turn a Unix program into a bare metal program." means.

Say all your program does is read some sensors via gpio, save those readings in ram, do a bit of math on that state, then set some other gpio pins, and that's all the entire system does. when it turns on, it should jump right into doing that and nothing else.

It may be easier to write that on a full sized Linux system, but come time to deploy it, all that Linux code that isn't being used is a liability for bugs and unexpected behavior to crop up. One approach is to slim down Linux to avoid that (eg removing systemd, esp for systems that don't have enough ram to run it), but another approach would be to just use something less complicated than Linux. It would suck to get to space only to find that something had messed up permissions on the gpio pins again so your satellite didn't deploy and now you've wasted a launch. If there's no permissions system to mess up, that can't go wrong.

As far as bare metal, I presume it means to port a Unix program which has all sorts of assumptions about running on a Unix operating system, to an OS that's not one, with simplified models for a lot of things, and for the system to reliably run that program on boot and not do anything unecessary. Like run cups.

For some tasks you don't need even need an OS and that competes with QNX. For instance I do hobby experiments with persistence of vision displays on AVR-8 and assembly language on the bare metal is so simple but it is simple hardware and simple software, bigger jobs need something more.

As it has been already said, with RTEMS and other similar real-time operating systems it is easy to predict the worst case time for the handling of any event. This is absolutely necessary when controlling many kinds of hardware devices.

This is completely impossible with standard Linux. Real-time Linux might offer some guarantees, but I am pretty sure that they are rather soft.

Linux is already too big to be easy to understand and audit in order to be certain that you really know which will be the worst case.

With a simple OS like RTEMS, a competent programmer can become an expert in its internals in a month or so, and then it is easy to understand what exactly will happen in any circumstance.

I have used in the past some commercial real-time operating systems, like pSOS and VxWorks, but the free RTEMS has been much better than those, especially because being open-source it allows a much deeper customization than any proprietary solution.

chibios is great, but I haven't followed their culture or licensing in a few years now.

my experiences then would lead me to recommend it, though.

ChibiOS doesn't really compete in the same space as QNX. It's a single address space executive that's more like FreeRTOS than QNX.

I believe seL4 is formally verified.

Verified worst case execution time analysis. So instead of just hanging on it until it achieves the desired responsiveness (like QNX and rt Linux) Sel4 can actually back up its claim.

In that case, you’re just hanging on the specs, prover, compiler, and other dependencies. Things most developers can’t even begin to verify. That also assumes the hardware works which isn’t safe if talking about cosmic rays.

Empirical testing of “verified” software often found errors, too. High-assurance, security certification used to require formal methods, thorough testing of code, review of anrtifacts, and pen testing by independent team. We should do that today.

In this case, you’d do white box and black box testing of the WCETS of seL4.

Agree but I have seen more people wrongly think they need hard real time than people incorrectly picking their kernel.

I don’t doubt this but it’s worth noting that I believe spacex uses Linux as their RTOS.

SpaceX can afford that by their _very_ resilient design and accepting, even anticipating failure.

If your design is resilient to missing deadlines, you don't need a RTOS and the question becomes moot.

talk nerdy to me, which ones ?

RT Linux is not the only alternative. There are now plenty of options for achieving hard real-time performance without relying on proprietary RTOS solutions:

  Dedicated Real-Time Cores: Many SoCs include dedicated cores specifically for hard real-time tasks.
  Microcontroller Companions: Pairing a microcontroller running a lightweight RTOS with a larger, general-purpose processor as a supervisor (You can use normal Linux).
  FPGAs: Custom FPGA implementations can deliver deterministic timing where ultra-low latency or specialized processing is needed.

And ofc, RT Linux, that in my experience is suitable for over 80% of hard real-time applications (an estimation, based on experience as a former user—your mileage may vary).

Out of curiosity, to what extent can modern x86 / Arm systems support hard-realtime guarantees?

I'm wondering about things that could cause unexpected delays, like PCIe bus contention / error correction, high interrupt traffic, etc.

(I don't know much about this topic, so apologies if my question is ill-conceived.)

If by "modern" you mean a desktop/server CPU, the problem is the complexity that exists because of optimization for average throughput. Do you really know how long you will wait, worst-case, if three different cores flush the same cache line back to DRAM? Or maybe some niche hardware you use has some funky behaviour that stalls some hardware bus, or waits in the kernel for a millisecond, twice a month or whatever.

On the other hand, on FPGAs, deterministic timing is so very simple. Your output will not be a single clock cycle late even if something else goes wrong in logic running on the same FPGA (except through a connection that you control).

If you really want to know, OSADL has a QA Farm that monitors worst-case interrupt and scheduling latency for various CPUs and Linux versions.

Real time is about worst case time, not average.

The classic test for a real time OS is simple. You write a program that waits from an interrupt from an input pin. When the input is raised, there's an interrupt. That activates a user process. The user process turns on an output pin.

You hook this up to a signal generator and a scope. Feed in a square wave. See the delay between input and output. If there are outliers, delays much larger than the normal delay, the system is doing real time properly at all.

If it passes that test, run some other program at a lower priority than the one that's monitoring the input pin. The results should not change. The other program should reliably be preempted.

QNX could pass that test, at least when I last used it.

Note that this requirement conflicts with many modern CPU and OS features, such as sleep modes, power-saving under light load, and paging. Classically, you didn't care, because the control CPU used far less power than whatever large piece of equipment it was controlling. But for things that run on batteries, it's a problem.

Something that's given trouble: "system management mode". Some industrial computers used code running in system management mode to help out with some peripherals, such as making flash memory look like a hard drive. This time-stealing shows up in the interrupt latency test. QNX techs used to quietly have a blacklist of embedded computers to avoid.

> If there are outliers, delays much larger than the normal delay, the system is doing real time properly at all.

Just to clarify for anyone reading, I think you meant:

> If there are outliers, delays much larger than the normal delay, the system is not doing real time properly at all.

That has to be the correct reading.