News for hackers. Stuff that matters.
Re:Undefined behavior is often a thing for C programmers (2011) by flohofwoe on Thursday September 03, 2020 @09:47AM (#24363985)
Unpopular opinion: the problem isn't so much undefined behaviour, but rather that compiler vendors have chosen to exploit undefined behaviour in their optimization passes.

PS: some pragmatic cleanup of UB and IB in the C standard would be welcome though.

Re:Undefined behavior is often a thing for C programmers (2011) by shultays on Thursday September 03, 2020 @10:02AM (#24364126)
The reason for having UB in the first place is so compilers can exploit them so we can have more optimized programs.

UB is a feature, not a bug. So you don't really "clean" it up. Cleaning up of UB means a lot of overhead for programs.

Re:Undefined behavior is often a thing for C programmers (2011) by geofft on Thursday September 03, 2020 @10:15AM (#24364257)
Yes, but also, UB is a "feature" to work around the fact that C is a relatively un-expressive language with a simple type system. It's great that the compiler can optimize

    for (i = 0; i < *len; i++)
        array[i] = 0;
without having to check on each iteration whether len secretly points inside the array and got modified by the loop. It would be better if this were not a heuristic and there were clear compile-time information saying for sure that this optimization is safe.
Re:Undefined behavior is often a thing for C programmers (2011) by sharpneli on Thursday September 03, 2020 @10:22AM (#24364337)
If len and array are both of the same type then the compiler must assume they can alias (or one of them is char*). One has to explicitly state with restrict that they cannot alias.

In case of similar but where it’s float array[N] etc then compiler can assume that they do not alias.

Re:Undefined behavior is often a thing for C programmers (2011) by geofft on Thursday September 03, 2020 @10:26AM (#24364385)
Right, that's a heuristic, and the consequence of that heuristic is that a bunch of things where you intentionally have pointers of different types to the same memory - the sort of low-level data manipulation that C ought to be good for - is UB unless you are careful about it and you find one of the exceptions to this rule.
Re:Undefined behavior is often a thing for C programmers (2011) by sharpneli on Thursday September 03, 2020 @10:35AM (#24364474)
Not a heuristic. Well defined, even if it could be defined better. I'd personally make all types not alias by default and then have a separate keyword 'alias' to specify things that must alias.

One of the reasons why Fortran was faster for decades was the default aliasing rules, restrict finally got us out of that trap. If we'd always assume everything alias plenty of code would become way slower.

Also C has well defined ways to access memory via different types. Make an union type and access via that. It makes it explicit.

Re:Undefined behavior is often a thing for C programmers (2011) by geofft on Thursday September 03, 2020 @01:09PM (#24366169)
It's deterministic, but it's still a heuristic in the sense that it's a guess at the underlying intent of the code. You can come up with different, perhaps simpler guesses, but they're still guesses.

It's quite common in existing real-world C code to rely on the ability to fill in a variable via a pointer of one type and then access it via a pointer of another type, and you expect the language to properly return the data you wrote to that pointer. The C committee determined that, just about always, that happens when one of the pointers is a char pointer (e.g., you call read() or write() on a struct), and they said that char pointers are allowed to alias by default but others can't. That's their guess, which is mostly accurate. There are much nicer answers here if you're not obligated to be compatible with existing C code - but in that case, especially today, I'd go with one of several other languages that can accurately express intention at the source code level instead of making backwards-incompatible changes to C. :)

Re:Undefined behavior is often a thing for C programmers (2011) by AlotOfReading on Thursday September 03, 2020 @10:47AM (#24364550)
Is it really fair to call type punning "well defined"? To my knowledge, it was considered implementation defined from C89 all the way through C11.
Re:Undefined behavior is often a thing for C programmers (2011) by sharpneli on Thursday September 03, 2020 @11:37AM (#24365055)
Before C11 char* was the only way to do type punning, badly. So it was basically forbidden, but still well defined.

Since then they specified that unions are the way to go.

Re:Undefined behavior is often a thing for C programmers (2011) by jzwinck on Thursday September 03, 2020 @10:23AM (#24364343)
"restrict" does exactly what you're looking for.
Re:Undefined behavior is often a thing for C programmers (2011) by rurban on Thursday September 03, 2020 @05:08PM (#24368928)
If it wouldn't be broken it would indeed be useful. With gcc restrict is still broken, you have define it away. clang had fixed it just recently I think. https://github.com/autoconf-archive/autoconf-archive/pull/20...
Re:Undefined behavior is often a thing for C programmers (2011) by bitwize on Thursday September 03, 2020 @11:01AM (#24364684)
The reason was originally to admit portability to different architectures and different environments. The primary difference between undefined behavior and implementation-defined behavior is that implementation-defined behavior must be accepted by the compiler and yield a working program. Undefined behavior can error out at compile or run time -- but it doesn't have to.

Some examples:

1. When C came out, you couldn't assume a two's complement machine. So some architectures would yield different results on signed over/underflow than others, and some architectures might trap on overflow. Hence, signed overflow is undefined in C.

2. Did you know the Lisp Machine had a C compiler? It did. Because pointers on the Lisp machine were tagged and bounds-checked to ensure memory safety, this compiler implemented C pointers with fat pointers, containing a reference to an object and an offset within that object. Most machines you are familiar with allow indirecting through arbitrary integers; the Lisp Machine is an example of one that does not, trapping on invalid pointer accesses. Having C and C programs available in both environments mean that indirecting through a pointer that does not point to or inside some pre-existing static or dynamically allocated object is undefined.

3. Operating systems with virtual memory can arrange things such that indirecting through 0 (NULL) will raise a segmentation violation in user code, guaranteeing a pointer value that is always invalid -- but inside the kernel you may want to access memory location 0. So accessing null pointers is undefined.

By leaving these undefined, you are not committing implementors to adapt the abstract machine to the underlying hardware in cases where hardware can widely differ, allowing for simple implementations that yield fast code on all architectures they compile for yet still allowing for a useful subset of C code to be 100% portable.

Re:Undefined behavior is often a thing for C programmers (2011) by kazinator on Thursday September 03, 2020 @04:19PM (#24368425)
Two's complement machines can trap on signed overflow, or set CPU flags that can be tested.

MIPS compilers for C emit "addu" instructions for signed integer additions rather than using "add", to avoid overflow traps.

Using unsigned arithmetic for signed is not just so that C programs relying on the behavior do not blow up. When overflow behavior is reversible wraparound, the compiler more freedoms in rearranging a calculation. Suppose the programmer has written some a + b + c, and ensured that there is no overflow (when it's added left to right, as the abstract semantics says). If the addition mechanism has reversible overflow, the compiler can rearrange it to do the additions in any order. The result will be the same, correct one that (a + b) + c would produce.

But the idea that two's complement itself "wraps around" is mostly a myth. Two's complement has a fixed range of representation, and a result of adding two elements from that representation can be out of that range. In that case, the truncated value whose bit pattern corresponds to the unsigned arithmetic is not the correct result.

Re:Undefined behavior is often a thing for C programmers (2011) by shultays on Thursday September 03, 2020 @03:31PM (#24367821)
You can force portability and remove UB at the same time. For example the system is not two's complement? just wrap everything math operation with checks on those system and say that your language is two's compliment!

Or just does what C does, make it UB and the programmers will be aware of implementation of overflows and won't suffer because of added checks. They will do the checks themselves if they want and handle their numbers with care

Imo the answer always boils down to performance. Java has no UB yet it works on billion devices (citation needed). It is doable, just costly.

Re:Undefined behavior is often a thing for C programmers (2011) by bitwize on Wednesday September 09, 2020 @06:19PM (#24426568)
No, it doesn't boil down to just speed. The goal is close correspondence between C source and the emitted assembly. This gets you a few advantages, one of which is performance up to a point (beyond which you will need to optimize). Another is ease of implementation, on a variety of architectures.
Re:Undefined behavior is often a thing for C programmers (2011) by kazinator on Thursday September 03, 2020 @04:21PM (#24368437)
C compilers use unsigned arithmetic even on hardware that offers built-in overflow traps for signed, two's complement arithmetic (no extra instructions required).
Re:Undefined behavior is often a thing for C programmers (2011) by flohofwoe on Friday September 04, 2020 @07:02AM (#24373836)
What I mean with "pragmatic cleanup" is getting rid of obsolete assumptions about the underlying hardware. E.g. two's complement has won, there's no reason why this couldn't be reflected in the C standard by removing signed overflow from the list of undefined behaviour. I bet more such cases can be found (e.g. 8-bit bytes, NULL not being zero, etc...).

Even if such exotic hardware is still around, the only thing that would change is that C compilers supporting this hardware couldn't call themselves "ISO Compliant" anymore, the users of those compilers wouldn't lose anything.

Re:Undefined behavior is often a thing for C programmers (2011) by shultays on Friday September 04, 2020 @10:04AM (#24375014)
C++20 is actually forcing two's complement I think. So it is happening slowly
Re:Undefined behavior is often a thing for C programmers (2011) by fanf2 on Thursday September 03, 2020 @02:34PM (#24367194)
The rationale does not mention optimization as a reason for undefined behaviour https://www.lysator.liu.se/c/rat/a.html#1-6
Re:Undefined behavior is often a thing for C programmers (2011) by shultays on Friday September 04, 2020 @03:57AM (#24372756)
Eh, well but it is. Since most (all?) UB could be eliminated if you added more code. I don't see any other reason for stuff like making access to uninitialized memory UB for example.
Re:Undefined behavior is often a thing for C programmers (2011) by pornel on Thursday September 03, 2020 @10:57AM (#24364646)
Compilers use UB for optimization, because programmers de-facto demand it.

You'd hate a compiler where this has silly overhead:

    for(int i=0; i < n; i++) arr[i];
That's because 64-bit addressing modes don't overflow the same way as signed 32-bit ints. But signed overflow being UB gives compilers license to ignore the difference and generate code that makes sense.

The real problem is that C is too bare-bones and lax to be able to contain or avoid UB in day to day programs. For the aforementioned overflow gotcha, there's no way to enforce use of size_t for indexing. There's no `checked_mul(a,b)` to easily avoid signed UB. You have to make the check yourself, and if you're not careful, that check will be UB too. malloc() makes you multiply numbers often, exactly where it's dangerous. And on top of that implicit integer conversions can make unsigned arithmetic surprisingly signed.

Re:Undefined behavior is often a thing for C programmers (2011) by esrauch on Thursday September 03, 2020 @11:25AM (#24364925)
Something that I haven't seen discussed is why it wouldn't be a better solution would be for the code you listed to be a warnings or errors; get the user to write size_t then it is efficient without needing to go to UB. Another common example is a null-check after a deref; it seems like that should also just be a compiler error since the UB optimization is based on "you wrote code that the compiler can just directly tell is wrong, so it's going to pretend you didn't do that".

There's surely cases where it would be hard to suggest a "equally efficient without UB" replacement (eg when it wants to fuse two i64 into one), but it feels like the current position is too far on the axis of "provided with suspicious code, the compiler should just make that garbage run as fast as possible and call it a day"

Re:Undefined behavior is often a thing for C programmers (2011) by dooglius on Thursday September 03, 2020 @11:11AM (#24364789)
Presumably you meant to do something with arr[i] or the whole loop could be elided. I'm not sure I understand your point, what is the type of n? If it's something bigger than int that's a bug and should result in a warning.
Re:Undefined behavior is often a thing for C programmers (2011) by aw1621107 on Thursday September 03, 2020 @11:41AM (#24365095)
From what I understand, the type of n isn't really the important part here; it's the type of the indexing variable (i) that makes a difference. On platforms where sizeof(int) is not the size of a register (e.g., x86_64 with 32-bit ints), UB signed overflow allows the compiler to not have to emit code to deal with the possibility of 32-bit overflow, which probably results in "sensible" assembly.

This article [0] goes into a bit more detail.

[0]: https://gist.github.com/rygorous/e0f055bfb74e3d5f0af20690759...

Re:Undefined behavior is often a thing for C programmers (2011) by dooglius on Thursday September 03, 2020 @12:05PM (#24365445)
The article you linked says that compilers can prove a lack of overflow in this example, so UB is not necessary to assume here. It also claims that the compiler cannot prove this in more complex examples (but gives no explicit examples). What I'm saying is if the compiler cannot prove a lack of overflow in this way, that should be a warning.
Re:Undefined behavior is often a thing for C programmers (2011) by aw1621107 on Thursday September 03, 2020 @01:37PM (#24366475)
> The article you linked says that compilers can prove a lack of overflow in this example, so UB is not necessary to assume here.

It's trivial to prove in that particular example because it's assumed that count is also an int (both i and count are in 32-bit registers).

> It also claims that the compiler cannot prove this in more complex examples (but gives no explicit examples).

The example doesn't even have to be that complex; you just need sizeof(n) > sizeof(i) and i to be signed (godbolt example at [0]). In that example, if the compiler can assume signed overflow is UB, then you get fairly sensible code generation. If the compiler cannot make that assumption (add the -fwrapv flag), then you get that movsxd instruction in the middle of the loop body. If you use -O2 instead of -O1, then -fwrapv prevents loop unrolling.

> What I'm saying is if the compiler cannot prove a lack of overflow in this way, that should be a warning.

I'm curious how frequently this warning might fire in real-life codebases, or when it does fire how frequently it's something the programmer can actually fix. Clang does have -Rpass-missed to emit some diagnostics when an optimization cannot be performed, but it doesn't emit anything in this particular case (though I'm not sure whether it's supposed to).

[0]: https://godbolt.org/z/abq3n9

Re:Undefined behavior is often a thing for C programmers (2011) by kps on Thursday September 03, 2020 @10:22AM (#24364339)
The problem is that the initial concept was botched by accident (C89) and then embraced rather than corrected by later committees.

Nobody during C89 formulation realized that it would end up “a license for the compiler to undertake aggressive optimizations that are completely legal by the committee's rules, but make hash of apparently safe programs” that “negates every brave promise X3J11 ever made about codifying existing practices, preserving the existing body of code, and keeping (dare I say it?) `the spirit of C.'” If they had, the response would have been that “[UB] must go. This is non-negotiable.”

Re:Undefined behavior is often a thing for C programmers (2011) by MereInterest on Thursday September 03, 2020 @10:32AM (#24364447)
You have many quotation marks, but no reference to who said them. Those aren't quotes from the articles, so I can't find where they came from.
Re:Undefined behavior is often a thing for C programmers (2011) by kps on Thursday September 03, 2020 @10:53AM (#24364605)
A letter to X3J11 from some chap named Dennis Ritchie.
Re:Undefined behavior is often a thing for C programmers (2011) by raphlinus on Thursday September 03, 2020 @11:02AM (#24364693)
For people who read the comments here but not the primary sources[0], you've put words into Dennis Ritchie's mouth. He did not say that UB must go, but that noalias must go. And, lo and behold, it did. So these (altered) quotes are not at all a fair characterization of what's actually in the standard.

[0]: https://www.lysator.liu.se/c/dmr-on-noalias.html

Re:Undefined behavior is often a thing for C programmers (2011) by kps on Thursday September 03, 2020 @11:07AM (#24364756)
He said why ‘noalias must go’, and those reasons apply equally to the modern conception of UB, so it's clear that the consequences of the definition were not realized at the time.
Re:Undefined behavior is often a thing for C programmers (2011) by raphlinus on Thursday September 03, 2020 @09:50AM (#24364012)
This is not an unpopular opinion per se, but one that puts you firmly in the "semi-portable C" camp (in the "camp" model of the essay I linked in another comment), while the compiler vendors themselves have moved entirely away from that camp into the "standard C" camp. As I wrote, "The principles and practices in one camp can seem alien, even threatening to another."
Re:Undefined behavior is often a thing for C programmers (2011) by chrisseaton on Thursday September 03, 2020 @09:50AM (#24364007)
This is the mainstream opinion.
Re:Undefined behavior is often a thing for C programmers (2011) by sharpneli on Thursday September 03, 2020 @10:24AM (#24364364)
With a nonoptimizing compiler the first example just segfaults at dereferencing P.

I’m not quite sure what the compiler is supposed to do in that case. Insert nullchecks by itself to avoid crashes?

Re:Undefined behavior is often a thing for C programmers (2011) by mastax on Thursday September 03, 2020 @09:53AM (#24364045)
How would you modify the standard and/or optimization passes for the first example in the article?
Re:Undefined behavior is often a thing for C programmers (2011) by k1t on Thursday September 03, 2020 @10:25AM (#24364374)
You can't remove A (redundant given B) and then subsequently remove B (redundant) as well.

As soon as your first pass decides to rely on the presence of the dereference as a reason for removing the null check, the dereference needs to be flagged as critical, such that it cannot be removed by a subsequent pass.

Better still, if the compiler does try to remove something that was relied on by an earlier optimization step, alert the user that something is probably wrong with the code.

Re:Undefined behavior is often a thing for C programmers (2011) by MereInterest on Thursday September 03, 2020 @10:40AM (#24364501)
Maybe this is me coming from a C++ background rather than C, but that sounds like a horrible idea. That would mean that any code removed by a branch determination (e.g. if(0) in a macro/template expression) would be assumed to still exist for the further purpose of removing unused variables.

Also, this would prevent one of the main benefits of function inlining. Once you have inlined a function, the values of some of the parameters may be known at compile time, resulting in lots of those if(0) parameters. But that is something that was relied on in previous passes, even though it is known to be a static value in the current pass.

Re:Undefined behavior is often a thing for C programmers (2011) by dooglius on Thursday September 03, 2020 @11:47AM (#24365184)
Why would it mean that? The compiler can still remove anything inside an if(0).
Re:Undefined behavior is often a thing for C programmers (2011) by microcow on Thursday September 03, 2020 @10:25AM (#24364373)
@mastax, you could model pointer dereferencing as an effect and require dereferencing null pointers to trap.
Re:Undefined behavior is often a thing for C programmers (2011) by btrask on Thursday September 03, 2020 @10:30AM (#24364429)
Fun fact: if there were no C language definition, there would be no "undefined behavior."
Re:Undefined behavior is often a thing for C programmers (2011) by steerablesafe on Thursday September 03, 2020 @10:41AM (#24364516)
In that case all C program behavior would be literally undefined.
Re:Undefined behavior is often a thing for C programmers (2011) by btrask on Thursday September 03, 2020 @01:39PM (#24366498)
Is that what you think of Python?

How can this be: C has a standard, when many other languages do not. Thus C is "more defined" than other languages. Yet somehow it ends up being less defined.

If C had no standard, it would be defined by its implementation, just like many other languages. But because it has a standard, it has undefined behavior.

The real trick is that without a standard, implementations have to be bug-compatible. They're too defined! The purpose of a standard is to define them less.

Re:Undefined behavior is often a thing for C programmers (2011) by steerablesafe on Thursday September 03, 2020 @01:56PM (#24366688)
Python many times had the problem of being defined as CPython. Modules that target multiple Python implementations often have to work around differences between them, without a standard they have to look into implementation details of their targets. Not having a standard is definitely a pain.

But for most intents and purposes Python is defined by its documentation, and it's fine.

Having said that UB in C and C++ means behavior that the standard doesn't define. If you remove the standard then UB either loses meaning or it makes all programs undefined. It definitely doesn't make all programs defined.

Re:Undefined behavior is often a thing for C programmers (2011) by btrask on Thursday September 03, 2020 @03:01PM (#24367491)
If you remove the standard, wouldn't everything naturally be implementation-defined? Like, say, Python.
Re:Undefined behavior is often a thing for C programmers (2011) by nwmcsween on Thursday September 03, 2020 @06:50PM (#24369741)
Yep and you would end up with what was occuring before there a standard was created (POSIX, etc) which is generally a giant dumpster fire of implementation defined hell.
Re:Undefined behavior is often a thing for C programmers (2011) by btrask on Thursday September 03, 2020 @10:18PM (#24371118)
Have you thought about why having a standard causes all these problems with undefined behavior? The point of a standard is to be less specific than a reference implementation. That's why it's written in English, not code. That means definitions are fuzzy, which is why every kind of English-language law, rule, or spec is vulnerable to malicious compliance.

If C had a judge, like the legal system, he could deny attempts to use fuzziness in the standard (intended to prevent specifying bugs) to introduce bugs. Unfortunately, C doesn't have a judge with common sense, it has people who think the standard is code and anything that is undefined really is a license to do whatever you want.

I guess this problem was inevitable as soon as the words "nasal demons" were put to keyboard. Now the only thing that can save us is a reference implementation.