News for hackers. Stuff that matters.
Re:Zig's Curious Multi-Sequence for Loops by jeroenhd on Monday February 27, 2023 @01:39PM (#34960214)
On many platforms, the zero page is set up so access to it will always segfault. This isn't a language guarantee, but it's a guarantee in most modern operating systems (Linux, FreeBSD, Windows). This is set up for pointers all the way up to the end of the first page.

On Windows and Linux this is the first 4KiB so range 0x0000 up to 0x1000, unless large pages are on (then it's even more).

On macOS in x64 this is the entire 4GiB memory space, probably a method to help developers port their 32-bit software to x64. I don't know what the zero page size on ARM is.

If your microcontroller doesn't have this guarantee, you can't make use of this feature.

Re:Zig's Curious Multi-Sequence for Loops by rom-antics on Monday February 27, 2023 @02:20PM (#34960697)
That's a guarantee on the level of the hardware/OS, but hardware semantics are not the same as language/compiler semantics. Even if according to the source code you're dereferencing a pointer value 0x0 or 0x2, that doesn't mean the compiler-emitted machine code will end up telling the hardware to do the same.

Remember this gem?

https://kristerw.blogspot.com/2017/09/why-undefined-behavior...

Once you trigger UB, all bets are off and your code could do anything. A segfault just means you spun the roulette wheel, bet it all on red, and got lucky your house wasn't bulldozed.

Zig also uses LLVM under the hood, right? So it's subject to these same semantics. An LLVM pointer value cannot legally contain arbitrary non-null non-pointer integers such as 0x2. That's a dead giveaway of UB. And I doubt the emitted Zig code safety-checks every pointer dereference for a value less than 0x1000 before performing the dereference.

Re:Zig's Curious Multi-Sequence for Loops by kristoff_it on Monday February 27, 2023 @03:17PM (#34961338)
> An LLVM pointer value cannot legally contain arbitrary non-null non-pointer integers such as 0x2.

0x2 is a perfectly valid pointer value, it just happens to never be a good virtual memory address on modern systems where virtual memory is setup by the usual OSs, hence the fact that you can rely on it segfaulting.

Re:Zig's Curious Multi-Sequence for Loops by jeroenhd on Monday February 27, 2023 @03:20PM (#34961370)
The semantics are actually operating system and even compiler flag dependent. On macOS you can choose the size of your zero page during build. The numbers I've listed are just the defaults.

Zig UB is not C UB. There is an entire language built on top of it. Just because something behaves a certain way in C, doesn't mean the same thing is true in Zig. Zig is no longer a code generator for C, it has switched to a self hosted compiler a while back. In fact, the language is rapidly progressing to the point where LLVM is a mere optional dependency.

I don't know the semantics around LLVM pointers. I don't see why 0x2 would be invalid, there are plenty of platforms programmed in C(++) that have a flat memory model. It would be quite painful to have a microcontroller where you can't send data to the output pin because LLVM decided that 2 is invalid (but 0 isn't). I've never seen LLVM complain about invalid dereferencing, though, it always ends up doing what the compiler tells it to do as far as I can tell.

Zig pointers will definitely cause UB but most Zig code shouldn't need them. Slices are actually bound checked and should probably be preferred in most cases of pointer arithmetic. Simple pointers can't be increased or decremented so you need to manually go through @intToPtr if you want to do real pointer arithmetic, which is quite unusable.

I haven't used Zig much so I don't know how many Zig semantics are copies of C semantics and how many are translated by the Zig frontend. However, "this is a bad/undefined thing in C so it must be a bad/undefined thing in Zig" is simply not true.

Re:Zig's Curious Multi-Sequence for Loops by rom-antics on Monday February 27, 2023 @04:28PM (#34962101)
I know Zig is not C, that's why I specifically mentioned LLVM. It's fine if Zig has different opinions about UB than LLVM does, but in that case ReleaseSafe builds should not use LLVM, not even optionally. If Zig says some operation is defined, but LLVM says it's undefined, well, LLVM is the one optimizing code so it's LLVM's invariants that matter. Right now it looks like Zig is playing fast and loose with correctness, shoving everything through LLVM but not respecting LLVM's invariants. And hey, if something is observed to segfault under some conditions today on the current version of LLVM, we'll just say segfaults are guaranteed. It's disappointing to see.
Re:Zig's Curious Multi-Sequence for Loops by AndyKelley on Monday February 27, 2023 @05:18PM (#34962765)
A lot of people have the same misunderstanding as you.

LLVM has rules about what is legal and what is not legal. If you follow the rules, you get well-defined behavior. It's the same thing in C. You could compile a safe language to C, and as long as you follow the rules of avoiding UB in C, everything is groovy.

Likewise, this is how Zig and other languages such as Rust use LLVM. They play by the rules, and get rewarded by well-defined behavior.

Re:Zig's Curious Multi-Sequence for Loops by rom-antics on Monday February 27, 2023 @05:53PM (#34963143)
Is not one of the LLVM rules, pointers must be valid and have a valid provenance in order to be dereferenced? If 0x2 ends up in a pointer that is dereferenced (or 0x0 in a nonnull pointer), has that rule not been broken? And if the rule is broken, does that not trigger undefined behavior?
Re:Zig's Curious Multi-Sequence for Loops by AndyKelley on Monday February 27, 2023 @06:21PM (#34963461)
I invite you to share a snippet from the LLVM language reference[1] that backs up your interpretation.

I will return the courtesy, with regards to my interpretation:

> An integer constant other than zero or a pointer value returned from a function not defined within LLVM may be associated with address ranges allocated through mechanisms other than those provided by LLVM. Such ranges shall not overlap with any ranges of addresses allocated by mechanisms provided by LLVM. [2]

[1]: https://llvm.org/docs/LangRef.html

[2]: https://llvm.org/docs/LangRef.html#pointer-aliasing-rules

Re:Zig's Curious Multi-Sequence for Loops by rom-antics on Monday February 27, 2023 @07:00PM (#34963858)
From the same section,

- Any memory access must be done through a pointer value associated with an address range of the memory access, otherwise the behavior is undefined.

- A null pointer in the default address-space is associated with no address.

A null pointer (0x0) is associated with no address, therefore it has no address range. So if you do attempt a memory access (dereference), the behavior is undefined. QED. A naive translation to assembly would indeed segfault on a modern OS, but LLVM's optimizations are free to assume that code path is unreachable and do anything else.

Once the program is in this state, a bug of some kind is unavoidable. I don't take issue with that - what I take issue with is your claim that this behavior is well-defined, because it definitely is not. It would be equally valid for a null dereference to corrupt your program state or wipe your hard disk.

Re:Zig's Curious Multi-Sequence for Loops by AndyKelley on Monday February 27, 2023 @07:18PM (#34964040)
You have already admitted that 0x1, 0x2, etc. are fine. Your remaining argument rests entirely on the incorrect premise that Zig's only option is to lower to LLVM IR using the default address space.
Re:Zig's Curious Multi-Sequence for Loops by rom-antics on Monday February 27, 2023 @07:52PM (#34964286)
I don't think 0x2 is a valid pointer either. The docs say the pointer value must be "associated with address ranges allocated through mechanisms..." - to me the word "allocated" means it's the result of an allocation, pointing at usable address space. (Sorry, I know this is a purely semantic argument. Debating the meaning of words does not make for very interesting discussion.)

In Rust for example, derefencing a raw pointer is unsafe - because that pointer could have a value of 0x2 - which would result in undefined behavior according to LLVM.

tbh I'm surprised any of this is even up for debate. If you google "is segfault undefined behavior" you'll get 100 results telling you yes, yes it is.

Re:Zig's Curious Multi-Sequence for Loops by anonymoushn on Tuesday February 28, 2023 @09:35AM (#34969709)
Are you claiming that any program that segfaults exhibits undefined behavior within LLVM semantics, even those that were not compiled by LLVM? Or within some other set of semantics shared by all programs that can segfault?
Re:Zig's Curious Multi-Sequence for Loops by rom-antics on Tuesday February 28, 2023 @08:12PM (#34977440)
I'm claiming that if a program is compiled with LLVM, it must follow must LLVM's rules. One of those rules is that a pointer must be valid in order to be dereferenced. If a program attempts to dereference an invalid pointer and segfaults, it has broken those rules* and thus exhibited undefined behavior. While undefined behavior MAY result in a segfault, it's equally valid for the program to continue running with corrupted state and wipe your hard disk in a background thread.

I'm not sure how I can connect the dots any more clearly. Like gggggp said, it's baffling to see the creator of a popular language sweep the nasal demons under the rug and pretend that certain undefined behavior is guaranteed.

Calling such segfaults "safe" or "well-defined" is setting your users up for disappointment and CVEs, because a "well-defined" result is axiomatically impossible in the presence of undefined behavior. It's subtle, and if we were talking about a Java competitor maybe I could forgive the mistake. But if you're writing a low-level language it's important to understand how this stuff works. Ironically, he spread misinformation in the very post where he accused Rust evangelists of the same.

*excluding something silly like `raise(SIGSEGV)`

Re:Zig's Curious Multi-Sequence for Loops by anonymoushn on Wednesday March 01, 2023 @02:20AM (#34979645)
Sure, I think I understand. The claim is maybe that it's legal for LLVM to emit code that (before every pointer access to a pointer obtained from outside of LLVM) somehow checks whether the pointer points to a region of memory that was actually allocated outside of LLVM and does different stuff based on the result of that check. In the face of such adversarial codegen on the part of LLVM, if someone wanted to implement this correctly on Linux, they might need to make sure they actually mapped the pages they wanted to use for crashing with PROT_NONE before using any pointers pointing into the crashing region. Is that right?

Do the docs actually define exactly which mechanisms external to LLVM count as allocating address ranges and which do not? It's possible that calling mmap and passing PROT_NONE does not count, for example.

Re:Zig's Curious Multi-Sequence for Loops by rom-antics on Wednesday March 01, 2023 @01:00PM (#34985821)
I wouldn't call codegen adversarial. The optimizer isn't out to get you. It emits the best code it can given a certain set of assumptions. It may just seem adversarial at times because the output can behave in unintuitive ways if you break those assumptions.

I don't believe PROT_NONE suffices. The address needs to be accessible, not merely mapped. If reading through a pointer, the address must be readable. If writing through a pointer, the address must be writeable. This is why writing to a string constant is undefined behavior, even though reading would be fine.

Another issue is alignment. If you read from a `*const i32` with unaligned pointer value 0x2, the optimizer is free to assume that code path is unreachable and, you guessed it, bulldoze your house. If you get a segfault from reading an `i32` from address 0x2, you've already hit UB and spun the roulette wheel.

In theory the emitted code could check pointers for alignment and validity (in whatever platform-specific way) before accessing them, and simulate a segfault if not. Such checks would serve as optimization barriers in LLVM, and prevent these instances of UB. Of course Zig's current ReleaseSafe doesn't do this, and I think it would be silly if it did. But that's the only way you could accurately call segfaults "well-defined".

Re:Zig's Curious Multi-Sequence for Loops by anonymoushn on Wednesday March 01, 2023 @09:19PM (#34991104)
I hope in the future we will get new LLVM semantics that are capable of expressing programs that use guard pages.
Re:Zig's Curious Multi-Sequence for Loops by avgcorrection on Monday February 27, 2023 @03:32PM (#34961502)
> On many platforms, the zero page is set up so access to it will always segfault. This isn't a language guarantee, but it's a guarantee in most modern operating systems (Linux, FreeBSD, Windows). This is set up for pointers all the way up to the end of the first page.

Then I guess it could be a language guarantee if Zig only supports/targets those platforms. However, considering how low-level Zig is, I doubt that that is the case.