News for hackers. Stuff that matters.
Re:OpenBSD now enforcing no invalid NUL characters in shell ... by semiquaver on Tuesday September 24, 2024 @10:08AM (#41636616)
I wonder if it’s https://justine.lol/ape.html / cosmopolitan libc
Re:OpenBSD now enforcing no invalid NUL characters in shell ... by tiffanyh on Tuesday September 24, 2024 @12:16PM (#41638140)
Just yesterday I asked @jart, here on HN, about Cosmo & OpenBSD.

https://news.ycombinator.com/item?id=41627889

Some interesting tidbits in the GitHub link provided in the HN comment above.

Re:OpenBSD now enforcing no invalid NUL characters in shell ... by chubot on Tuesday September 24, 2024 @12:07PM (#41638056)
I'm pretty sure it is, I remember reading something about this

Yeah I found it here

https://news.ycombinator.com/item?id=41030960

2019 bug - https://austingroupbugs.net/view.php?id=1250

https://justine.lol/cosmo3/

> This is an idea whose time has come; POSIX even changed their rules about binary in shell scripts specifically to let us do it.

FWIW I agree with this OpenBSD change, which says more pointedly

All the shells are written in C, and majority of them use C strings for everything, which means they cannot embed a NUL, so this is not surprising. It is quite unbelievable there are people trying to rewrite history on a lark, and expecting the world to follow alone.

i.e. it's not worth it to change a bunch of old code in order to allow making code more esoteric.

We want systems code to be more predictable, reliable, and less esoteric ... not more esoteric

Re:OpenBSD now enforcing no invalid NUL characters in shell ... by asveikau on Tuesday September 24, 2024 @03:25PM (#41640003)
> POSIX even changed their rules about binary in shell scripts specifically to let us do it.

I'd seen this quote around. The fact that the standards were changed to allow it never struck me as a good indication that it should be relied upon. It seems rather backwards of how these standards work.

I got flamed on HN once for saying cosmopolitan libc shouldn't be used for production because it relies on weird behaviors and implementation quirks that aren't really an ABI.

Re:OpenBSD now enforcing no invalid NUL characters in shell ... by comex on Tuesday September 24, 2024 @04:16PM (#41640451)
Looking at this further, the standards change doesn't even match what Cosmopolitan is doing.

From the 'changed their rules' link, the 'Desired Action' is to add this text: "The input file may be of any type, but the initial portion of the file intended to be parsed according to the shell grammar [..] shall not contain the NUL character."

This handles things like shar archives where you have a shell script at the beginning, then an exit command, then binary gunk.

But Cosmopolitan binaries are not just shell scripts with binary. They're hybrids of shell script and DOS executable. And apparently this requires putting nul bytes right near the beginning (see my other comment, https://news.ycombinator.com/item?id=41640331), in the "portion [..] intended to be parsed according to the shell grammar". Which explicitly violates the new text.

I can understand why this hack is needed for what Cosmopolitan is trying to accomplish, but it makes no sense to claim POSIX blessed it.

Re:OpenBSD now enforcing no invalid NUL characters in shell ... by chubot on Tuesday September 24, 2024 @04:22PM (#41640506)
Yeah exactly, that was my reading too! The claim in the link doesn't match what the POSIX bug says

If it did, then would be a sign that the POSIX process is not working well

Because POSIX is supposed to be descriptive of what exists, not prescribe new behavior

Re:OpenBSD now enforcing no invalid NUL characters in shell ... by enriquto on Tuesday September 24, 2024 @04:30PM (#41640591)
> POSIX is supposed to be descriptive of what exists, not prescribe new behavior

Well, it can be argued that ape already existed when that POSIX change was written :)

Everything is working as intended. Nothing to see here. Move along. Move along.

Re:OpenBSD now enforcing no invalid NUL characters in shell ... by eesmith on Tuesday September 24, 2024 @12:07PM (#41638051)
Shouldn't be. See the "exit 1" in your link? That's the end of the shell script, and as the OpenBSD link says;

> It remains possible to put arbitrary bytes AFTER the parts of the shell script that get parsed & executed (like some Solaris patch files do). But you can't put arbirary bytes in the middle,

Re:OpenBSD now enforcing no invalid NUL characters in shell ... by oguz-ismail on Tuesday September 24, 2024 @12:20PM (#41638169)
It is. Binaries generated by cosmocc have NUL in the middle.
Re:OpenBSD now enforcing no invalid NUL characters in shell ... by comex on Tuesday September 24, 2024 @04:05PM (#41640331)
Ah, indeed. Here are the first 16 bytes of one:

4d 5a 71 46 70 44 3d 27 0a 00 00 10 00 f8 00 00 |MZqFpD='........|

There are already nul bytes here, and there are a lot more before the single quote gets closed at offset 0x200.

Re:OpenBSD now enforcing no invalid NUL characters in shell ... by eesmith on Tuesday September 24, 2024 @04:54PM (#41640795)
And I can confirm a NUL in 11th byte of my hello.c a.out:

  >>> s[:11]
  b"MZqFpD='\n\n\x00"
Looking closer, I missed the content of "BIOS BOOT SECTOR".