The embedded world is very slow to change, so you can read about "modern approaches" (i.e. approaches used today) in any book about embedded programming written in the last 30 years.

I currently work on spacecraft flight software and the only real advance on this project over something like the space shuttle that I can point to is that we're trying out some continuous integration on this project. We would like to use a lot of modern C++ features, but the compiler for our flight hardware platform is GCC 4.1 (upgrading to GCC 4.3 soon if we're lucky).

Having worked on embedded systems for a decade at this point, the fact that we allow vendors to get away with providing ancient compilers and runtimes is shameful. We know that these old toolchains have thousands of documented bugs, many critical. We know how to produce code with better verification, but just don't push for the tools to do it.

Isn't the key part that these older systems have documented bugs?

Or, to put it another way, if there's a wasp in the room (and there always is), I'd want to know where it is.

That doesn't end up being the case for a number of reasons. Firstly, no one is actually able to account of all of these known issues a priori. I don't like calling things impossible, but writing safe C that avoids any compiler bugs is probably best labeled as that.

Secondly, vendors make modifications during their release process, which introduces new (and fun!) bugs. You're not really avoiding hidden wasps, just labeling some of them. If you simply moved to a newer compiler, you wouldn't have to avoid them, they'd mostly be gone (or at worst, labeled).

Are the newer compilers truly that much better? I've been working in tech since the 90's, and I can't say that for the tools I've used I've noticed any great improvement in overall quality- bugs get swatted, new ones created in what feels like a constant measure. I am assuming that many optimizations are turned off regardless, due to wanting to keep the resulting assembly as predictable as possible, but I do not work in the embedded space, so this is perhaps a naive question.

I think the idea is that you don't want a whole wasp nest. Just a bunch of stray wasps.

I wonder if the same is true of Space X?

Yeah. AFAIK they use FreeRTOS for the real deeply embedded stuff which would look very familiar to this discussion.

If you don’t mind me asking, how could one get into this field if they’re already an experienced software engineer in the more “vanilla” stuff (web services, etc.)?

How do you do CI/CD for embedded systems?

CI during the first phases of development in my experience is now often done with modern tooling (gitlab CI, Jenkins), compiling and running tests on a regular Linux x86 build server. Later phases switch over to some sort of emulated test harness, with interrupts coming from simulated flight hardware. Obviously the further along in the development process, the more expensive and slow it is to run tests. Maybe some software groups (SpaceX?) have a process that allows for tight test loops all the way to actual hardware in the loop tests.

I can't speak for what the rest of the industry does, but some chip manufacturers provide decent emulators, so you can run some tests there. We have also done some hardware tests where we connect our hardware to a raspberry pi or similar and run our CI there. It doesn't replace real-world testing, but it does get us some of the way there.

I find it interesting that such critical code is written in C. Why not use something with a lot more (easily)statically provable properties. Like Rust or Agda?

You’ll find that for very serious, industrial applications, a conservative mindset prevails. C may not be trendy at the moment, but it powers the computing world. Its shortcomings are also extremely well known and also statically analyzable.

Also, think about when flight software started being written. Was Rust an option? And once it came out, do you expect that programmers who are responsible for millions of people’s lives to drop their decades of tested code and development practices to make what is a bet on what is still a new language?

What I find interesting is this mindset. My conservativeness on a project is directly proportional to its importance / criticality, and I can’t think of anything more important or critical than software that runs on a commercial airplane. C is a small, very well understood language. Of course it gives you nothing in terms of automatic memory safety, but that is one tradeoff in the list of hundreds of other dimensions.

When building “important” things it’s important to think about tradeoffs, identify your biases, and make a choice that’s best for the project and the people that the choice will affect. If you told me that the moment anyone dies as a result of my software I would have to be killed, I would make sure to use the most tried-and-true tools available to me.

> Also, think about when flight software started being written. Was Rust an option?

It wasn't, but Ada probably was (some flight software may have been written before 1980?), and would likely also be a much better choice.

> I can’t think of anything more important or critical than software that runs on a commercial airplane.

Nuclear reactors?

Arguably, the existence of nuclear reactors which don't fail safe under any contemplated crisis is a hardware bug. It's possible to design a reactor that can be ruptured by a bomb or earthquake, which will then dump core into a prepared area and cool down.

This kind of physics-based safety is obviously not possible for airplanes.

What triggers the core dump? Humans? Software? Are there detectors integrated into the walls?

Physical rupture of containment.

If all electronics fry simultaneously, then the reactor core cools in-place.

I should have said “commercial airplanes are among the most important and critical things that use software.” It’s obviously difficult to determine the objective most important use case.

Ada has existed for 40 years. This directly means it has nothing to do with being conservative.

And how many people know Ada vs. C? Orders of magnitude more right?

I think that’s the problem here - it’s important to analyze orders of magnitude accurately. C isn’t a little more conservative than Rust or Ada. It is orders of magnitude more conservative.

You're advocating throwing baby out with bathwater.

Rust interops with C seamlessly, doesn't it? You don't have to throw out good code to use a better language or framework.

C may be statically analyzable to some degree, but if Rust's multithreading is truly provable, then new code can be Rust and of course still use the tried and true C libraries.

Disclaimer: I still haven't actually learned any Rust, so my logic is CIO-level of potential ignorance.

The issue is that you’re trading a problem space that is very well understood for one that isn’t. Making a safe program in C is all about being explicit about resource allocation and controlling resources. So we tend to require that habit in development. It’s socialized. The only thing you’d be doing is using technology to replace the socialization. And you’d be adding new problems from Rust that don’t exist in the C world.

It’s tempting in a lot of cases to read the data sheet and determine that the product is good enough. But there are a lot of engineering and organizational challenges that aren’t written in the marketing documents.

Those challenges have to be searched for and social and technological tools must be developed to solve those challenges.

As an exercise in use of technology it looks easy but there’s an entire human and organizational side to it that gets lost in discussions on HN.

> Rust interops with C seamlessly, doesn't it?

From someone who works in a mixed C + Rust codebase daily (Something like 2-3M lines of C and 100k lines of Rust), yes and no. They're pretty much ABI compatible, so it's trivial to make calls across the FFI boundary. But each language has its own set of different guarantees it provides and assumes, so it's easy to violate one of those guarantees when crossing a FFI boundary and triggering UB which can stay hidden for months.

One of them is mutability: in C we have some objects which are internally synchronized. If you call an operation on them, either it operates atomically, or it takes a lock, does the operation, and then releases the lock. In Rust, this is termed "interior mutability" and as such these operations would take non-mutable references. But when you actually try that, and make a non-mutable variable in Rust which holds onto this C type, and start calling C methods on it, you run into UB even though it seems like you're using the "right" mutability concepts in each language. On the rust side, you need to encase the C struct inside of a UnsafeCell before calling any methods on it, which becomes not really possible if that synchronized C struct is a member of another C struct. [1]

Another one, although it depends on how exactly you've chosen to implement slices in C since they aren't native: in our C code we pass around buffer slices as (pointer, len) pairs. That looks just like a &[T] slice to Rust. So we convert those types when we cross the FFI boundary. Only, they offer different guarantees: on the C side, the guarantee is generally that it's safe to dereference anything within bounds of the slice. On the rust side, it's that, plus the pointer must point to a valid region of memory (non-null) even if the slice is empty. It's just similar enough that it's easy to overlook and trigger UB by creating an invalid Rust slice from a (NULL, 0) slice in C (which might be more common than you think because so many things are default-initialized. a vector type which isn't populated with data might naturally have cap=0, size=0, buf=NULL).

So yeah, in theory C + Rust get along well and in practice you're good 99+% of the time. But there are enough subtleties that if you're working on something mission critical you gotta be real careful when mixing the languages.

[1] https://www.reddit.com/r/rust/comments/f3ekb8/some_nuances_o...

> On the rust side, it's that, plus the pointer must point to a valid region of memory (non-null) even if the slice is empty.

Do you have a citation for that, because it seems obviously wrong[0] (since the slice points to zero bytes of memory) and I'm having trouble coming up with any situation that would justify it (except possibly using a NULL pointer to indicate the Nothing case of a Maybe<Slice> datum)?

0: by which I mean that Rust is wrong to require that, not that you're wrong about what Rust requires.

Well the docs have this to say [1]:

`data` must be non-null and aligned even for zero-length slices. One reason for this is that enum layout optimizations may rely on references (including slices of any length) being aligned and non-null to distinguish them from other data. You can obtain a pointer that is usable as data for zero-length slices using NonNull::dangling().

So yes, this requirement allows optimizations like having Option<&[T]> be the same size as &[T] (I just tested and this is the case today: both are the same size).

I'm not convinced that it's "wrong", though. If you want to be able to support slices of zero elements (without using an option/maybe type) you have to put something in the pointer field. C generally chooses NULL, Rust happens to choose a different value. But they're both somewhat arbitrary values. It's not immediately obvious to me that one is a better choice than the other.

[1] https://doc.rust-lang.org/std/slice/fn.from_raw_parts.html

> [1] https://doc.rust-lang.org/std/slice/fn.from_raw_parts.html

Thanks.

> having Option<&[T]> be the same size as &[T]

That is literally what I mentioned as a possible reason ("except possibly ..."), but what I overlooked was that you could take a mutable reference to the &[T] inside a Option<&[T]>, then store a valid &[T] into it - if NULL is allowed, you effectively mutated the discriminant of a enum when you have active references to its fields, violating some aspect of type/memory safety, even I'm not sure which.

> C generally chooses NULL, Rust happens to choose a different value.

It's not about what pointer value the langauge chooses when it's asked to create a zero-length slice, it's about whether the language accepts a NULL pointer in a zero-length slice it finds lying around somewhere.

Wanting to suddenly start using rust would mean putting any and all tools through a tool qualification process, which is incredibly time consuming and vastly expensive. In the field of safety critical software, fancy new languages are totally ignored for, at least partially, this reason. What's really safer, a new language that claims to be "safe" or a language with a formally verified compiler and toolchain where all of your developers have decades of experience with it and lots of library code that has been put through stringent independent verification and validation procedures, with proven track record in multiple other safety critical projects?

Rust's official documentation on FFI ( https://doc.rust-lang.org/nomicon/ffi.html ) recommends using an external crate 'libc' to facilitate even the minimal FFI functionality. This crate is not part of Rust itself. It is apparently maintained by some of Rust's developers, but again, this is not an official Rust component. To me this does not seem like the kind of mature design you would rely on for interoperability with other languages.

Actually, Rust's std itself depends on that same libc crate, so it's a bit hard to say it's "not part of Rust itself".

Uh... Am I misunderstanding something here, doesn't that just make the situation even more dire?

Rust/C interop still has major challenges. It isn't seamless.

> Disclaimer: I still haven't actually learned any Rust, so my logic is CIO-level of potential ignorance

And yet you seem to write with such confidence. /Are/ you a CIO? It’s the only thing that makes sense.

Using a newer language carries a lot of risks and challenges for embedded programs:

- There’s a high risk of bugs in the compiler/standard library in languages with lots of features

- Usually, the manufacturer of an embedded platform provides a C compiler. Porting a new compiler can be a LOT of work, and the resulting port can often be very buggy

- Even if you can get a compiler to work, many newer languages rely on a complicated runtime/standard library, which is a deal-breaker when your complete program has to fit in a few kilobytes of ROM

I think the answer was right there in their comment. "The compiler for our flight hardware platform is GCC 4.1 (upgrading to GCC 4.3 soon if we're lucky)".

Often, the only high-level language available for an embedded platform is a standard C compiler. If you're lucky.

Ada is used a fair amount in high $ projects. Toolchains are expensive, and the C compiler is provided for free from the chip / board vendor.

Because safety critical fields are also slow-moving.

Isn't Ada already used in areospace industry?

> Where can someone (i.e., in my case a software engineer who's working with Kotlin but has used C++ in his past) read more about modern approaches to writing embedded software for such systems?

The JPL coding guidelines for C [1] are an amusing, first-hand read about this stuff. Not sure if you would qualify them as "modern approaches".

[1] https://en.wikipedia.org/wiki/The_Power_of_10:_Rules_for_Dev...

I can testify first-hand that the "functions in a single page" and "avoid the pre-processor" rules are not followed very closely haha

> A minimum of two runtime assertions per function.

I am guessing the idea is to catch runtime errors in the test phase, and assertions are disabled for the production build.

Searching for things like "MISRA C" and "real-time computing" will help you get started.

Thanks a lot for the keywords; these are very good starting points to look for further stuff on the topic!

Didn't know that there was a term (i.e., real-time computing) for this kind of systems / constraints.

I'd also look a the Joint Strike Fighter C++ Coding Standard. Stroustrup himself hosts it as an example of how C++ is a multi paradigm language that you can use a subset of to meet your engineering needs.

http://www.stroustrup.com/JSF-AV-rules.pdf

My older co-workers have some great alternative definitions of that initialism.

If you want a "toy" example of this type of code, look at flight control software for drones such as Betaflight. You can modify this code and test it in real life. I did this, as I contributed the GPS Rescue feature. I have a blooper reel of failures during testing.

https://github.com/betaflight/betaflight/

Just read docs that were written in the 70s, before the advent of garbage collection.

Garbage Collection is from 1959, though - and Unix & C's original model pretty much matches "bump allocate then die" with sbrk/brk and lack of support for moving.

Fully static allocation is the norm though for most "small" embedded work.