News for hackers. Stuff that matters.
Re:Defensive Programming by tyhoff on Wednesday December 23, 2020 @02:37PM (#25521247)
Author here. Firmware, for better and worse, is stuck in C/C++ land, so many of the topics here are essential to keep a complex, multi-MCU, multi-board setup in working order.

I'm actually really curious what you all do in C/C++ to prevent bad operations from ever being performed in the first place.

For example, should we just change our internal malloc / free to use double pointers instead?

  bool malloc(size_t n, void **ptr) {
    *ptr = <memory>
  }

  void free(void **ptr) {
    ... <free>
    *ptr = NULL 
  }
This way, if anyone actually tries to use that pointer, it will crash the system (hardfault), instead of potentially using corrupted memory, which IMO is much worse.
Re:Defensive Programming by dragontamer on Wednesday December 23, 2020 @04:39PM (#25522535)
> I'm actually really curious what you all do in C/C++ to prevent bad operations from ever being performed in the first place.

1. Go as far as you can with static memory. Don't call the allocator: its slow, sometimes requires inter-thread communications.

2. Dynamic uses often can use the stack: almost always in L1 of your local core. Stack allocations are self-cleaning, just don't pass those pointers to anyone else.

3. unique_ptr<blah> covers most of the true dynamic issues.

4. shared_ptr<blah> covers the rest of them.

---------

The above advice is just general-purpose / low-performance code.

If you're entering high performance coding (aka: you're actually worried about fragmentation, memory sizes, and such), then things get more complicated. Embedded is often RAM-restricted, so you end up in this complicated place where you need to think of highly-efficient techniques.

That's almost always a complexity / resource tradeoff. Efficient techniques are just innately more complex than general purpose techniques. Try not to reach for efficient coding unless you know its something you need to do

Re:Defensive Programming by saagarjha on Wednesday December 23, 2020 @08:19PM (#25524232)
> Dynamic uses often can use the stack

But they should also be very careful when deciding when to use the stack. Uncontrolled input + alloca = a bad time.

Re:Defensive Programming by ffffjfjjfj on Wednesday December 23, 2020 @03:07PM (#25521590)
This gives a false sense of security, since the most common use-after-free bugs are if you kept a pointer around. Your method doesn't stop things like this:

    void *p1, *p2;
    malloc(100, &p1);
    p2 = p1;
    ...
    free(&p1);
    *p2; // oops!
Obviously its trivial in an example like this, but when p2 leaves the lexical scope then there's a problem.
Re:Defensive Programming by btrask on Thursday December 24, 2020 @04:16AM (#25526115)
If you are transferring ownership, you would do

    p2 = p1; p1 = NULL;
However if you are intentionally doing multiple ownership, then yes you can still have problems.
Re:Defensive Programming by sgtnoodle on Thursday December 24, 2020 @03:41AM (#25525972)
For MCU code, it's best practice to statically allocate all memory. If you do need dynamic allocation, it's better to use either fixed size pool allocators, or allocate only pools.

For performance critical code running on a more powerful system with an OS, it's still good to stick with static allocation, or malloc and free only during initialization time.

Over the last decade I've written firmware for space ships, self driving cars, and autonomous aircraft, and I've never needed more than an allocate only pool or an O(n) LRU cache data structure. I've written plenty of software that was hard on the heap, but rarely does memory need to be freed during runtime, and it's rare to need raw pointers in C++ when writing such code.

Ideally, your "firmware" is written in a way that abstracts away the target hardware enough that it's basically just weird software. Then, you can unit test it, and even run linters and memory sanitizers on it.

Re:Defensive Programming by lmm on Wednesday December 23, 2020 @07:33PM (#25523961)
> This way, if anyone actually tries to use that pointer, it will crash the system (hardfault), instead of potentially using corrupted memory, which IMO is much worse.

Oh you sweet summer child. Null pointer deference is undefined behaviour, the compiler will happily turn it into corrupting memory under the right circumstances.

Re:Defensive Programming by tyhoff on Wednesday December 23, 2020 @08:01PM (#25524125)
> Oh you sweet summer child

Should have clarified. On the embedded systems I work on, generally Cortex-M4, you set an MPU region on address 0x0 and it turns into a hardfault, so it's pretty easy to catch.

Re:Defensive Programming by lmm on Wednesday December 23, 2020 @10:14PM (#25524744)
If you were writing assembly that would be reliable, but in C it isn't. The compiler knows that NULL dereferences are undefined behaviour and so won't necessarily compile them into memory accesses.
Re:Defensive Programming by btrask on Thursday December 24, 2020 @04:38AM (#25526192)
Yeah, in C you need to use assertions (or simple checks) for things that might be null.

That said the compiler isn't infinitely smart (thank god) and complex null derefs will "safely" make it to runtime. (What a sad world we're in.)

Re:Defensive Programming by saagarjha on Wednesday December 23, 2020 @08:20PM (#25524239)
A good optimizing compiler can remove the dereference entirely if it notices, unfortunately.
Re:Defensive Programming by sigjuice on Wednesday December 23, 2020 @08:23PM (#25524257)
How big is this MPU region? Pointers to large structs can easily allow access past the protected region.
Re:Defensive Programming by elygre on Thursday December 24, 2020 @12:58PM (#25529725)
It is undefined by the standard, a compiler is thus free to make null pointer references into hard faults. You need to know your tool chain and your environment.
Re:Defensive Programming by tutfbhuf on Thursday December 24, 2020 @06:25AM (#25526624)
Why is it stuck in the C/C++ land? I think if you can write a firmware in C++, then it should be possible to write it in rust, too. Am I wrong?
Re:Defensive Programming by steveklabnik on Thursday December 24, 2020 @08:38AM (#25527322)
It is very possible to write firmware in Rust, as long as the device is using a chip that’s supported.
Re:Defensive Programming by mkoubaa on Wednesday December 23, 2020 @06:02PM (#25523307)
One thing I try to do in c++ is pass things by reference in lower level libraries so that client code has to be defensive but library code assumes non null.
Re:Defensive Programming by _0ffh on Wednesday December 23, 2020 @05:25PM (#25522983)
There is at least the option to use a more modern language if it translates to C.

Nim for example does that, and has compiler options for embedded systems.

https://nim-lang.org/docs/nimc.html#nim-for-embedded-systems

Re:Defensive Programming by curtis3389 on Wednesday December 23, 2020 @12:47PM (#25520076)
In Erlang, instead of calling it Offensive Programming they call it "Let it Crash".

A very valuable concept either way: if a function encounters an error and can't somehow resolve it and still do its work successfully, then it should fail and not do something else.

Re:Defensive Programming by pjbk on Wednesday December 23, 2020 @03:45PM (#25522007)
Based on my experience, many, many, many headaches would be saved if this advice was followed more closely.

IMHO defensive techniques including recovery should be placed in runtime monitors and supervisors, not in underpinning code which should have a minimum set of behaviors. It should also be kept as simple as possible to avoid code bloat and help reviewers and tools like static analyzers do their job.

Old computing hardware was designed like that. Then software attempted to replicate hardware "exceptions", however not realizing that the model was intrinsically different and completely exploding in intricacies dealing with interrupts, stacks, memory management and threads.

In a similar way, supervisors should be as decoupled as possible from the target code to provide reliability and avoid overall system complexity, with their own set of narrowed down requirements.

Sadly our industry is plagued with historical, ad-hoc programming techniques that are not relevant anymore or were plain empirical bad advice in first place. For example, blindly relying on stateful paradigms, or side effects by design or as an acceptable consequence of certain programming idioms.

Re:Defensive Programming by macintux on Wednesday December 23, 2020 @01:52PM (#25520750)
Agreed. Among the other fantastic things about Erlang, the fact that = is effectively an assertion means that unlike other languages where you turn off assertions in production, Erlang retains them in every environment.
Re:Defensive Programming by alpaca128 on Thursday December 24, 2020 @04:17AM (#25526119)
And it's valuable in other languages as well. Not many things are as frustrating as silent failures in some library. And it's still one of my biggest gripes with JavaScript that when it encounters an error there's a good chance it just keeps on running in some undefined state, hiding the root cause of the problem.
Re:Defensive Programming by mauvehaus on Wednesday December 23, 2020 @01:34PM (#25520554)
I've worked in both embedded and in the Windows file system stack.

What's unsaid in this article is that your ability to break noisily varies a lot based on what the end user's expectations are and how much of the device you own.

If you're building a single-purpose embedded device, sure, you can probably halt and catch fire if its use-case allows doing so safely. This might involve making sure the outside world is safely brought to a halt, but that might be feasible if you're building a device that works on a production line.

On the other hand, you almost certainly don't want to bugcheck some random big corporate file server that might be doing goodness knows what else to get a crash dump. At least not without the blessing of the client.

Like so many other things in engineering, how to handle errors depends greatly on the use case, the users, and the maintainers (which might or might not overlap with the users). You can't baldly claim that one approach or another is best without understanding those.

Re:Defensive Programming by uxp100 on Wednesday December 23, 2020 @12:53PM (#25520144)
As a side note, this article suggests using assert. Fine, but make sure you know what your assert does, and when it does that, if you have a debug and release build (which I think you probably shouldn't.)

I've spent some time with "offensive" programming in the context of both security (like key leakage) and safety (like firmware integrity). It was just kinda funny thinking about meetings spent determining what we could do if i2c or something wasn't responding, and different approaches to try and fix it, and then a year later going back and saying, ok, if something isn't working, we've got an easy answer. Lockdown and halt boot. Anything seems weird, just stop trying.

Re:Defensive Programming by karlding on Wednesday December 23, 2020 @02:21PM (#25521064)
> make sure you know what your assert does, and when it does that, if you have a debug and release build

Absolutely. Probably the most common difference I've seen in "Debug" and "Release" builds relying on the standard library headers is when NDEBUG is defined as a macro when <assert.h> is included, which causes assert to become a no-op [0].

This has interesting implications when developers accidentally rely on the side effects of functions and assert on the return value of that helper. On "Release" builds that side effect would be compiled out. For that case, clang-tidy has the bugprone-assert-side-effect [1] checker if you want to enforce this in your CI.

[0] https://en.cppreference.com/w/c/error/assert

[1] https://clang.llvm.org/extra/clang-tidy/checks/bugprone-asse...

Re:Defensive Programming by greatgib on Wednesday December 23, 2020 @01:13PM (#25520327)
It's easy and a lot of manufacturer do the 'unexpected circonstances? Stop and reboots.

But it is really nocive in the context of real products. This is how you finish with the samsung blu ray player that became suddenly a brick. Unusable and unupdatable to be fixed.

Also, there used to be a lot of Android phone that became unusable bricks because they were stuck in boot loops and nothing could be done to go to a recovery or update stage.

That case was common for example when there was a few corrupted or dead bits in internal ram or storage.

Re:Defensive Programming by astrobe_ on Wednesday December 23, 2020 @02:24PM (#25521095)
Yes, it can be really counter-productive if done wrong. You have to do everything possible to reach a stage where a firmware update is possible. Only after that you can kick the reset line when you're unhappy.
Re:Defensive Programming by hansvm on Wednesday December 23, 2020 @01:48PM (#25520710)
There are _still_ android phones that become unusable bricks because they're stuck in boot loops. Google will happily send you another new phone, but the 4XL is vulnerable to one of the recent updates.
Re:Defensive Programming by iso8859-1 on Wednesday December 23, 2020 @01:11PM (#25520307)
Runtime assertions are easily inserted, but why not change the types, make that case impossible? Dependent types can make that happen.

Static assertions are enabled by dependent typing. But unlike dependent typing, static assertions are only assertions, for example they do not enable you to write a function that returns different types based on its arguments. Static assertions do not allow you to write a compile-time vector type.

Re:Defensive Programming by AnimalMuppet on Wednesday December 23, 2020 @01:56PM (#25520795)
"Making that case impossible" is hard for i2c not responding. You can represent "not responding" in a way that isn't confused with real data (even if zero-length), and you can refuse to go down the path that expects data. But no dependent type can make it respond. And you're still stuck with the questions: "What do I do then? How do I handle it? Can my program continue at all? How do I tell the user what's wrong?" As far as I can see, dependent types don't help with the larger problem at all.
Re:Defensive Programming by lmm on Wednesday December 23, 2020 @07:30PM (#25523939)
Types (you don't even need full dependent types) let you check your reasoning about why something is true, which is IME the core of the larger problem. Most of the time the bug doesn't happen because everyone forgot that it's possible for i2c to not respond, it's because component a thought that it should pass the i2c response straight through to component b and component b would handle the i2c not responding, whereas component b thought that component a had already handled the case where the i2c didn't respond (or rather, component b is a pass-through middleware that calls component c that wraps component d and component d thought that i2c not responding had already been handled). Types absolutely help with that.
Re:Defensive Programming by hansvm on Wednesday December 23, 2020 @02:13PM (#25520973)
Plausibly, there could be plenty of cases for favoring runtime assertions over dependent types. Or for mixing and matching even in cases where one or the other would probably suffice.

- Dependent types aren't easily available everywhere. If you don't have them then you can't use them.

- The compiler can't prove some statements (admittedly not a big issue for the examples from TFA). Supposing you have a dependent type system which allows you to state that a property holds, it could still be prudent to have a runtime check. If the type system doesn't allow such statements then it isn't sufficiently expressive to solve the problem in the first place.

- Assertions are a local change which one can make incrementally, whereas changing the types being used will immediately require larger changes (if the types are used more than once) or necessitate some extra wrapper code.

Re:Defensive Programming by greatgib on Wednesday December 23, 2020 @12:53PM (#25520143)
The arguments of this article are ridiculously bad in my opinion.

Defensive programming does not mean that you ignore errors. It means that the dev should always expect the worse and decide right away how to deal with it early.

Imagine you have a car that suddenly block and explode each time an unexpected error is encountered. So that you are sure to catch it.

So, you are driving on the highway, then a there a window fuse that blow and because of that your car decide to brake and explode...

Especially in hardware and embedded, you have to expect the unexpected to happen.

Imagine the car will suddenly stop when a lightbulb break or there is a shortcut in windows control. Sure you will know, but you will have 300 000 cars all around the world bricked!

Re:Defensive Programming by tyhoff on Wednesday December 23, 2020 @02:30PM (#25521171)
Author here! I don't think we are in disagreement, just mostly in different interpretations. You are talking about a car, with likely 200-300 embedded systems backed into it.

If the light sensor or the braking module goes into a completely unexpected state where it can no longer be communicated with, it has corruption, etc, you need to do something about it.

The solution isn't to reboot the car, it's to reboot the individual module and quickly get it back into working order. Rebooting the braking or lighting module should take on the order of milliseconds (if done correctly), and that is much, much better than having a piece of software in a corrupted state for seconds/minutes/hours.

In a car, there is a mothership board that is orchestrating everything, pinging every module, and ensuring everything is in working order. Just like a kubernetes setup, if any one piece fails to ping or looks off, just reboot it and get it working ASAP.

Re:Defensive Programming by uxp100 on Wednesday December 23, 2020 @01:01PM (#25520231)
I believe "offensive" programming is heavily used in automotive. If there is an issue with the connection to the camera, it is better to show nothing than corrupted and potentially misleading images. Better not to try to guess about uncertain throttle position.

Defensive programming can sometimes look like silently ignoring errors. A haphazard test of defensively written code (that, for example falls back to a different encryption scheme or io device) could pass incorrectly. But there is a place for defensive programming too. I think most important is being aware of which is which.

Re:Defensive Programming by theamk on Wednesday December 23, 2020 @05:35PM (#25523071)
Well, this is still better than just keep accelerating if there is a bug in ECU module. I would take “engine dies” over “stuck throttle” anytime.
Re:Defensive Programming by andi999 on Wednesday December 23, 2020 @12:35PM (#25519949)
Well... in standard linux malloc never fails... not sure if it is a good thing though..
Re:Defensive Programming by mcguire on Wednesday December 23, 2020 @01:03PM (#25520245)
Not really, no.

My first experience with that behavior was with AIX in the early '90s: allocation always succeeds (unless you've exhausted your address space) because it is backed by virtual memory. If you do run low on physical memory, every process gets a SIGDANGER and is expected to free up memory. If that fails, the kernel starts sending SIGKILLs.

Unfortunately, exactly none of the normal utilities understood the protocol and upon running low, the kernel just started shooting processes in the head until the situation improved. For some reason, one of the first processes chosen was usually inetd, meaning you could no longer remotely log in to see what was going on.

When I complained, I was told that the machines were frequently used for scientific computing, where they would allocate giant arrays which were then used sparsely, so it was the intended behavior. And that the X server at the time made use of the behavior.

So, ... yech.

Re:Defensive Programming by q3k on Wednesday December 23, 2020 @01:31PM (#25520526)
It does though?

    $ uname -a
    Linux mimeomia 5.9.11 #1-NixOS SMP Tue Nov 24 12:39:15 UTC 2020 x86_64 GNU/Linux
    $ cat main.c
    #include <stdio.h>
    #include <stdlib.h>
    
    int main() {
        printf("malloc: %lx\n", malloc(128LL << 30)); // 128GiB
        perror("malloc err");
    }
    $ gcc main.c -o main && ./main
    malloc: 0
    malloc err: Cannot allocate memory
Re:Defensive Programming by dragontamer on Wednesday December 23, 2020 @01:40PM (#25520619)
The original poster is almost certainly talking about Linux's overcommit policy: http://opsmonkey.blogspot.com/2007/01/linux-memory-overcommi...

You can't adopt a defensive / offensive stance with regards to out-of-memory issues, because on Linux... malloc doesn't fail when you run out of physical memory.

Your "error" occurs on a pointer-dereference, which could be anywhere in your program. Once more memory is dereferenced than the system has capacity for, the OOM-killer is activated. There's no defensive stance possible, aside from setting overcommit to "Type2" (disabling the overcommit behavior of the Kernel).

But that's somewhat non-standard, and possibly non-helpful, since some applications depend on overcommit.

Re:Defensive Programming by theamk on Wednesday December 23, 2020 @05:40PM (#25523122)
Disabling overcommit is totally feasible on some embedded boxes. And if you don’t want this for some reason, you can also mlock/madvice/manually touch all the memory to get errors right away as well.

Yes, overcommit makes things tricker, but it is hardly “impossible”

Re:Defensive Programming by dragontamer on Wednesday December 23, 2020 @06:19PM (#25523419)
> you can also mlock/madvice/manually touch all the memory to get errors right away as well.

All this does is run the OOM killer, which doesn't necessarily kill your process. Maybe its a process you didn't care about, maybe its a process you actually are communicating with and suddenly you're now in a weird deadlock state. Etc. etc.

Even if YOU touch all your memory like you're supposed to, there's no guarantee that OTHER processes have touched their memory.

So you touch memory unnecessarily: which means you're now officially reserving RAM (wasting it). Then, some OTHER process touches their memory and runs the OOM-killer. That wouldn't have happened if you just didn't touch your RAM.

-----------

If Linux Overcommit is on default, you live with it and die with it. No point trying to fight OS-policy. Change the policy (aka: Overcommit to 2), and then carefully select applications to run with that follow the same overcommit policy as you.

Re:Defensive Programming by andi999 on Friday December 25, 2020 @02:30AM (#25534918)
Exactly. The misunderstanding a lot of ppl have about the OOM killer is, is that the process touching memory which cannot be fulfilled will be killed. This is not the case (for all practial reasonings one shd assume it kills random processes)
Re:Defensive Programming by theamk on Friday December 25, 2020 @03:15PM (#25538717)
If you have an idea where do OOMs come from, you can (and should!) use "oom_score_adj"/"oom_adj" so that OOM kills processes are not random. There are many interesting strategies here:

- Set your watchdog and logging to "never kill". Set watchdog to reboot on OOM, after uploading error report. This way, if the system OOMs, you can log the problems to your logging server, and then watchdog will reboot the system to recover.

- If you have a "UI" process which can also eat all the memory, set it to "kill me first". Yes, the UI will crash, but the rest of system will restart it, and your updater would still work.

- If you have separate resource-intensive process, you can set it to "kill me first" -- then the rest of the system can stay up and maybe even show a descriptive error message.

- The latter approach works pretty nicely for general systems. Is your OOM killer activating because your build system started too many gcc instances? run it with "kill me first" setting -- it inherits to children processes, so you only build processes would be killed, the rest of the system (UI, browser, etc..) would all stay alive.

Re:Defensive Programming by andi999 on Monday December 28, 2020 @05:16AM (#25558073)
Thanks, this is interesting advice.
Re:Defensive Programming by anordal on Thursday December 24, 2020 @06:48AM (#25526726)
Excellent writeup and intro to the underdiscussed topic of offensive programming, and how DefP and OffP are not contradictory (any more than graceful fallback and rejection of bugs has to be), but rather applies to different categories of errors, with that crucial line of defense inbetween.

http://en.wikipedia.org/wiki/Offensive%20programming

Re:Defensive Programming by Geminidog on Wednesday December 23, 2020 @12:49PM (#25520095)
I was a python programmer before I became a c++ developer and believe it or not despite lack of type checking, big python apps have much less bugs then a comparable c++ app due in no part to offensive programming features built into python. In other words you don’t need to offensively program in python, it’s a given.

Firmware is just a pain in the ass to deal with. Wish there was a better way.

Re:Defensive Programming by ben-schaaf on Thursday December 24, 2020 @02:21AM (#25525668)
> believe it or not despite lack of type checking, big python apps have much less bugs then a comparable c++ app

Despite the lack of static typing, python has memory safety and a GIL; eliminating a much larger class of bugs, so it's not that hard to believe.

Re:Defensive Programming by Geminidog on Thursday December 24, 2020 @09:41AM (#25527827)
Additionally clear error messages similar to offensive programming. Any runtime error in python reads like a compile time error with the interpreter giving you the exact location where the error occured and what error occured.

You'd be surprised how many c++ programmers think python is a terrible language just simply due to the lack of type checking and indentations.

A lot of them have no idea how much easier and how much more productive life is outside of the c++ world. Take a look at the other replier. He's thinking "perhaps" it's true, he has no idea.

Re:Defensive Programming by rutthenut on Thursday December 24, 2020 @07:10AM (#25526840)
Firmware requires serious design, code and test regimes to deliver safe, and reliable components.

That same pain-in-the-arse requirement is true of other software, albeit higher-level platforms can perhaps save your arse when bugs are encountered

Re:Defensive Programming by Geminidog on Thursday December 24, 2020 @09:59AM (#25527972)
No. This shows a very narrow regime of experience. Testing is not the only way to verify software additionally the error surface area of certain languages is simply much much smaller in most languages outside of c++.

This had nothing to do with a language saving your arse. This has everything to do with the simple non-existence of a type of bug in other languages, for example memory safety in rust and python.

Additionally c++ often simply does not tell you where an error occurs and you have to use a coredump to figure everything out. It's not an offensive programming strategy by default. Python on the other hand makes every single type of error read like a c++ syntax error giving you a line number and an error type by default.

Another thing to consider is raw speed and lower memory requirements. These are the only advantages of c++. If your hardware isn't strict on the above requirements consider micropython.

I'm finding c++ programmers have this very narrow and cocky view of the world. They turn their nose up at languages like python without truly experiencing how much life is easier and more productive in those worlds.

Re:Defensive Programming by rutthenut on Monday January 04, 2021 @12:30PM (#25634759)
Far from narrow experience, C/C++ is just one of the languages I have used to deliver systems over a long period of time.

Not being cocky and not turning my nose up at Python as I know it can be productive and allow quick creation of solutions.

But pointing out that my experience has also encompassed high-resilience, high-availability, low-latency, fault-tolerance, high-security, air-gapped and embedded systems along the way. For the latter of those especially, the rigour of defining requirements and proving them during design and test phases, are key facets that go beyond the choice of any programming language whether modern or old.