News for hackers. Stuff that matters.
Re:The C++ Iceberg by flohofwoe on Wednesday May 08, 2024 @03:41AM (#40295362)
The UB mess only started when compiler writers decided it is ok to exploit UB for optimisations which then leads to wildly unpredictable behaviour when you actually hit UB.

But also: turn warnings to 11, use UBSAN, ASAN, TSAN and static analysers. These are all easy steps to make C and C++ code a lot more robust.

Re:The C++ Iceberg by H8crilA on Wednesday May 08, 2024 @04:08AM (#40295531)
No, UB has always been just undefined. `gets()`, one of the most exploited functions in the history of computers, does not require any clever malice from the compiler. And there are a lot of such examples. In fact practice has shown that it has always been very, very hard to write a program that cannot reach UB.

Thinking that you know what `undefined` really is results in undefined behavior.

Re:The C++ Iceberg by flohofwoe on Wednesday May 08, 2024 @04:27AM (#40295631)
Just because 'undefined behaviour' means 'theoretically anything can happen' doesn't mean that it will happen in practice. You also don't need to summon the UB ghosts to delete all your files, a regular logic bug works just as well, even in a language without any UB.

In older compilers, UB behaviour was a lot more predictable because the code transformations done by optimizing compilers were by far not as complex as today.

In the C standard, 'undefined behaviour' is basically a catch-all phrase for a lot of different things: from badly designed stdlib function that miss obvious security checks (like gets()) to the compiler generating non-sensical code.

The latter is what's much more dangerous. It's trivial to just not use the C stdlib functions, but code generation problems caused by UB are much harder to catch.

Re:The C++ Iceberg by H8crilA on Wednesday May 08, 2024 @04:33AM (#40295670)
No, it's the former that's more dangerous, and it has been there from the beginning. Every time you dereference a bad pointer (already free'd, corrupted, badly casted, ...), every time you go over the bounds of some array, every time you read memory that hasn't been initialized, every time you (write) access the same memory from two threads without synchronization, etc. - this is the bread and butter of binary exploitation. Exploits are fully expected under C/C++ spec, after all the behavior is undefined.
Re:The C++ Iceberg by flohofwoe on Wednesday May 08, 2024 @04:41AM (#40295717)
But at the same time it is totally expected in C and C++ that range checking is the responsibility of the programmer, and most of those problems are trivial to catch with Valgrind or ASAN. E.g. yes, it is a problem, yes, 70% of all CVEs are caused by memory corruption, but the problem is totally obvious and trivial to debug. It's basically part of the contract when deciding to use C or C++.

Compared to such 'obvious' memory corruption problems caused by invalid pointers or missing range checks, obscure code generation issues caused by UB are much harder to identify and the behaviour may also change randomly between compiler versions - this is the actually scary thing about UB, and it's also a relatively "new" thing.

Re:The C++ Iceberg by H8crilA on Wednesday May 08, 2024 @05:00AM (#40295807)
The point is exactly that "it is totally expected", and it has caused massive losses over the decades. It has been abused for everything ranging from turning your friend's PC off, to ransomware, to state-on-state espionage and even causing nuclear incidents in Iran's enrichment facilities.

The behavior is undefined.

Re:The C++ Iceberg by flohofwoe on Wednesday May 08, 2024 @05:14AM (#40295893)
Sheesh, the sky didn't fall, the world didn't end, and humanity survived. UB won't end the world. Relax.
Re:The C++ Iceberg by H8crilA on Wednesday May 08, 2024 @06:22AM (#40296314)
Perhaps we're not in a discussion that's worth continuing, but I'm talking about programming language design. There are languages with UB, and there are languages without UB - and I think in 2024 the verdict is clear.
Re:The C++ Iceberg by flohofwoe on Wednesday May 08, 2024 @07:28AM (#40296757)
Maybe from your point of view. But from elsewhere UB can be a useful tool because it allows wiggle room for compilers to support a wide range of platforms without sacrificing efficiency. It doesn't have to be as crazy as in modern C/C++ compilers, but some sane UB definitely makes sense (even Rust has that inside unsafe blocks).

One could also argue that most UB in C/C++ should actually be IB, but that's a different discussion.

Re:The C++ Iceberg by tialaramex on Wednesday May 08, 2024 @10:30AM (#40298515)
This "Wiggle room" would be at most implementation defined and in some cases just unspecified values, these probably aren't desirable but yes they're less dangerous than UB.

Rust has UB if you don't obey the rules (Safe Rust doesn't because the rules are not your responsibility in Safe Rust, the unsafe superpowers allow you to do things the enforcement can't check but now all rules are your responsibility) but it's not somehow magically "more sane" than C++ it's ultimately lowering to the same LLVM IR that Clang uses. It's tolerable because you're only confronting the risk when you explicitly choose to do so. If you don't need to squeeze that last few drops of performance you can leave it. If you don't have the confidence, if your experts are too busy to review it, if you haven't had a proper night's sleep, you can write safe Rust today.

Re:The C++ Iceberg by lolc on Wednesday May 08, 2024 @10:53AM (#40298749)
> the problem is totally obvious and trivial to debug.

We live in parallel universes that must have intersected in this thread.

Re:The C++ Iceberg by flohofwoe on Wednesday May 08, 2024 @12:56PM (#40300309)
Debugging tools got a lot better in the last two decades, memory corruption issues and leaks are by far not as scary anymore as in the 90s and early 2ks, for instance check out the realtime memory debugging panel in Visual Studio that's been there since VS2015. Also as mentioned at the top of the thread: compiler warnings, ASAN, UBSAN, TSAN and static analyzers are essential these days when writing C or C++ code.

The only barrier is to actually know about and use those tools in the development workflow.

Rust is still superior of course (for memory safety at least), because it actually prevents most of those issues in the first place, but it's not like the rest of the world has been twiddling thumbs in the meantime.

Re:The C++ Iceberg by Leherenn on Wednesday May 08, 2024 @06:03PM (#40303169)
It certainly got better, but it's still not trivial. With the sanitizers, you need to have code that actually triggers the UB, coverage is not enough. If your UTF-8 parsing code UB only on some very specific input, then unless your tests or own environment happen to have this input, you'll be none the wiser.

Fuzz testing helps, but it's not commonly used in my experience.

Re:The C++ Iceberg by Xeamek on Wednesday May 08, 2024 @04:15AM (#40295567)
It's not like UB is some magic functions that gets called when encountering UB.

Every UB could verry well be wrapped in error checking and we could have 0 UB out there. We don't do that, but that's because of (our) choice, not some superficial entity that poses machine when running into UB.

Re:The C++ Iceberg by gpderetta on Wednesday May 08, 2024 @04:27AM (#40295630)
> Every UB could very well be wrapped in error checking

Not in C and C++. How would you check that a pointer is safe to dereference for example?

Re:The C++ Iceberg by Xeamek on Wednesday May 08, 2024 @04:36AM (#40295687)
What do you mean? Just check if it's not null?
Re:The C++ Iceberg by gpderetta on Wednesday May 08, 2024 @04:55AM (#40295777)
Not all non-null pointers are safe to dereference.
Re:The C++ Iceberg by Xeamek on Wednesday May 08, 2024 @05:01AM (#40295814)
Like?
Re:The C++ Iceberg by flohofwoe on Wednesday May 08, 2024 @05:29AM (#40295997)
On the hardware and assembly level, all unmapped memory regions are illegal to access and will cause a segfault.

In high level languages it gets a lot more complicated (look up "pointer provenance").

Re:The C++ Iceberg by planede on Wednesday May 08, 2024 @05:36AM (#40296025)
Like one that is free'd or point to a stack object that is no longer in scope. Or one that points one past the last element of an array.
Re:The C++ Iceberg by Xeamek on Wednesday May 08, 2024 @05:42AM (#40296065)
It's unsafe to do that, but it's not UB
Re:The C++ Iceberg by planede on Wednesday May 08, 2024 @06:01AM (#40296184)
C23:

> The unary * operator denotes indirection. If the operand points to a function, the result is a function designator; if it points to an object, the result is an lvalue designating the object. If the operand has type "pointer to type", the result has type "type". If an invalid value has been assigned to the pointer, the behavior of the unary * operator is undefined.

below in a note (emphasis mine):

> Among the invalid values for dereferencing a pointer by the unary * operator are a null pointer, an address inappropriately aligned for the type of object pointed to, and the address of an object after the end of its lifetime.

Now the note is not normative, but I assume there are normative wording for defining what the "invalid" pointer values are, scattered around in the standard.

C++:

https://eel.is/c++draft/expr.unary.op#1.sentence-4

C++ is very particular in what it means for a pointer to point to an object, and this is also UB there.

Re:The C++ Iceberg by flohofwoe on Wednesday May 08, 2024 @05:56AM (#40296151)
I don't have all ca 200 UBs in my head (see https://gist.github.com/Earnestly/7c903f481ff9d29a3dd1), but I'm pretty sure all types of illegal memory accesses count as UB.

For instance from skimming the list:

- An object is referred to outside of its lifetime (6.2.4).

- The value of a pointer to an object whose lifetime has ended is used (6.2.4).

- An array subscript is out of range, even if an object is apparently accessible with the given subscript (as in the lvalue expression a[1][7] given the declaration int a[4][5]) (6.5.6).

Re:The C++ Iceberg by planede on Wednesday May 08, 2024 @06:10AM (#40296240)
Technically the array subscript example is UB even before the dereference. a[1][7] is equivalent to `*(a[1] + 7)` and a[1] + 7 itself is UB.

(6.5.7) Additive operators

> If the pointer operand and the result do not point to elements of the same array object or one past the last element of the array object, the behavior is undefined.

Interestingly a[1][5] is also UB, but for the dereferencing a past-the-end pointer, not for the arithmetic.

Re:The C++ Iceberg by gpderetta on Wednesday May 08, 2024 @06:18AM (#40296279)
Yes, that's exactly what I meant.

Also, dereferencing a pointer with the wrong dynamic type.

> all ca 200 UBs in my head

Technically there is no enumerated set of all possible UB. Anything not explicitly defined in the standard is UB.

/extremely-pedantic

Re:The C++ Iceberg by tialaramex on Wednesday May 08, 2024 @10:39AM (#40298616)
Effort to try to write an appendix listing all the UB and IFNDR is an ambition of WG21 but there's no specific timeline.

It's a bit like that "Migrate off obsolete DB server" card that's been sat in your team's (well most teams) backlog for a couple of years. Everybody agrees it should be done... But like not this sprint.

My assessment is that at their current rate WG21 is adding new UB to the language slightly faster than it's being documented for the appendix, so this won't actually finish.

Re:The C++ Iceberg by mpweiher on Wednesday May 08, 2024 @05:12AM (#40295880)
This turns out not to be the case.

Compilers have been known to remove exactly that sort of error checking code, as there was undefined behavior present.

It's a mess.

Re:The C++ Iceberg by Xeamek on Wednesday May 08, 2024 @05:22AM (#40295952)
Yes, I know it's not possible for a specific c++ dev to do such checks, as compilers can optimize them away.

What I meant is “we (as in) the c/c++ community (or rather the specification committee)” could tomorrow decide that, lets say, all null pointer dereferences should lead to program exit. Existence of UB is a choice.

Re:The C++ Iceberg by bigcheesegs on Wednesday May 08, 2024 @06:19AM (#40296287)
UB is a choice in that we could require symbolic execution of of all code. This is incredibly slow, so we don't.

There are many cases of UB that would be cheap to check, but there are many more that are incredibly expensive to check.

Re:The C++ Iceberg by mpweiher on Wednesday May 08, 2024 @05:10AM (#40295868)
> No, UB has always been just undefined

This turns out not to be the case.

There actually is text in the current standard as to what acceptable behavior is when encountering undefined behavior.

This was turned into a "note" in later versions of the standard, but initially it was just as much a normative part of the standard as everything else.

The text is still there.

Compiler writers want to exploit UB, hence they say that this text effectively doesn't exist.

https://blog.metaobject.com/2018/07/a-one-word-change-to-c-s...

Re:The C++ Iceberg by H8crilA on Wednesday May 08, 2024 @06:26AM (#40296338)
What's the behavior of this piece of code according to the spec? You can choose any C/C++ spec:

   void f() {
      char x[100];
      gets(x);
   }
Re:The C++ Iceberg by gpderetta on Wednesday May 08, 2024 @07:02AM (#40296589)
The key here is the meaning of "ignoring" in "ignoring the situation completely with unpredictable results".

Most programmers (and pretty much all compiler writers) interpret it as meaning that the compiler can assume it can't happen (ignore the possibility) and translate accordingly.

Some, like mpweiher interpret it as meaning that the compiler must implement a trivial (but somehow unspecified) translation to the underlying hardware and let the behavior be what the hardware provides.

edit: We went through this many many times.

Re:The C++ Iceberg by tialaramex on Wednesday May 08, 2024 @09:57AM (#40298193)
This craziness (a belief that if you don't specify what something means it must have some "obvious" behaviour that is coincidentally always whatever the programmer wanted) is endemic in C and C++ communities.

Right now, notionally serious people are making the case that C++ should enshrine the "bag-of-bits" model of pointers which is exactly this sort of stupidity. At first it looks like they want pointers are addresses which is kinda dumb but whatever, and you keep reading and they're like aha, also I should be able to dereference my bag-of-bits and have that work too... Full blown Provenance Via Integers, expect to spend the next forty years trying to specify how any of the optimisations you rely on can survive the resulting tangle, say hi to anybody who wanted Consume ordering and is still waiting.

Re:The C++ Iceberg by marssaxman on Wednesday May 08, 2024 @02:00PM (#40301101)
> This craziness (a belief that if you don't specify what something means it must have some "obvious" behaviour that is coincidentally always whatever the programmer wanted) is endemic in C and C++ communities.

That belief is endemic because that is the way things actually worked for many years! Compilers used to give us the obvious behavior, because they weren't smart enough to outfox us by doing anything else, because the CPUs we used were not fast enough to let them do the kinds of comprehensive optimization analyses which are now routine.

Re:The C++ Iceberg by gpderetta on Wednesday May 08, 2024 @03:12PM (#40301813)
No. The reality is that every new generation of programmers is completely comfortable with optimizations that existed before it started programming and surprised by optimizations added after that. Today we take for granted SSA, extensive interprocedural optimizations, value-range propagation and LTO. Before that was aggressive inlining and SRA; before that was smart register allocation and stack optimizations. Even before that it was instruction scheduling.

Each one of these optimizations have broken someone's code.

Re:The C++ Iceberg by mpweiher on Wednesday May 08, 2024 @12:59PM (#40300344)
The idea that "undefined behavior can't happen" is ludicrous, because it happens all the time. The idea that most programmers also believe this nonsense is entirely unsupported.

It is very popular in a small range of very influential companies.

Re:The C++ Iceberg by skitter on Wednesday May 08, 2024 @08:59AM (#40297614)
Compilers don't claim that part of the standart doesn't exist, they just go eith option 1: Ignoring the situation. E.g. if one branch of an if-statement is UB, compilers ignore that that branch can happen and only need to compile the branch they know exists.
Re:The C++ Iceberg by mpweiher on Wednesday May 08, 2024 @12:54PM (#40300279)
No, that's not ignoring the situation. Ignoring the situation is compiling the code and performing whatever operation happens, for example an out of bounds access.
Re:The C++ Iceberg by dwattttt on Wednesday May 08, 2024 @09:27AM (#40297882)
> Possible undefined behavior ranges from ignoring the situation completely with unpredictable results, to behaving during translation or program execution ...

As much as I'm not a fan of the status quo, the argument that changing "Possible" back to "Permissible" isn't particularly solid; the rest of the paragraph makes it clear it's not an exhaustive list of behaviours, merely examples from a "range".

Re:The C++ Iceberg by mpweiher on Wednesday May 08, 2024 @12:55PM (#40300291)
Yes, I agree that there are better solutions, this one is just minimal.

The "range" clearly indicates that you can do something that's semantically in-between the options provided.

What current compilers do is nowhere near that range.