laurenth

A shell is almost always used to setup the bootstrap environment, so the dependency on a shell is more or less always there.

Otherwise, something special with POSIX shell is its large number of independent implementations, making it the ideal starting point for diverse double-compilation (https://arxiv.org/abs/1004.5534). The idea is to bootstrap a toolchain from multiple compilers (shells in this case), and the result compared to verify that no shell introduced a trusting trust attack.

Author here,

> assuming the resulting shell script is as inscrutably as binary executable

It's quite the opposite, pnut generates shell code that's close to the original C code to make it easy to audit the code. A useful way to see pnut is as a tool that rewrites C code to POSIX shell, without significantly changing the structure.

This means that even if GCC is required for the initial compilation of pnut (GCC compiles pnut, then pnut compiles itself and we get the pnut-sh.sh script), the script can be "sanitized" from trusting trust attacks by simply comparing the script to the C code and making sure GCC hasn't introduced any malicious code.

Page 10 of the SLE24 presentation has a tombstone diagram showing the compilation steps to go from pnut's C code to a GCC binary: https://github.com/udem-dlteam/pnut/blob/main/doc/presentati...

We haven't found this to be an issue for Pnut. One of the metric we use for performance is how much time it takes to bootstrap Pnut, and dash takes around a minute which is about the time taken by bash. This is with Pnut allocating around 150KB of memory when compiling itself, showing that Dash can still be useful even when hundreds of KBs are allocated.

One thing we did notice is that subshells can be a bottleneck when the environment is large, and so we avoided subshells as much as possible in the runtime library. Did you observe the same in your testing?

That's the idea!

As you point out, it moves the trust from the binary to the shell executable, but the shell is already a key piece of any build process and requires a minimum level of trust. The technique of bootstrapping on multiple shells and comparing the outputs is known as Double Diverse Compiling[0] and we think POSIX shell is particularly suited for this use case since it has so many implementations from different and likely independent sources.

The age and stability of the POSIX shell standard also play in our favor. Old shell binaries should be able bootstrap Pnut, and those binaries may be less likely to be compromised as the trusting trust attack was less known at that time, akin to low-background steel[1] that was made before nuclear bombs contaminated the atmosphere and steel produced after that time.

0: https://dwheeler.com/trusting-trust/ 1: https://en.wikipedia.org/wiki/Low-background_steel

You're right, thanks for the bug report. It should now be fixed :)

It seems ShellCheck errs on the side of caution when checking arithmetic expansions and some of its recommendations are not relevant in the context they are given. For example, on `cat.sh`, one of the lines that are marked in red is:

  In examples/compiled/cat.sh line 7:
    : $((_$__ALLOC = $2)) # Track object size
      ^-- SC1102 (error): Shells disambiguate $(( differently or not at all. For $(command substitution), add space after $( . For $((arithmetics)), fix parsing errors.
      ^-----------------^ SC2046 (warning): Quote this to prevent word splitting.
        ^--------------^ SC2205 (warning): (..) is a subshell. Did you mean [ .. ], a test expression?
                   ^-- SC2283 (error): Remove spaces around = to assign (or use [ ] to compare, or quote '=' if literal).
                     ^-- SC2086 (info): Double quote to prevent globbing and word splitting.

It seems to be parsing the arithmetic expansion as a command substitution, which then causes the analyzer to produce errors that aren't relevant. ShellCheck's own documentation[0] mention this in the exceptions section, and the code is generated such that quoting and word splitting are not an issue (because variables never contain whitespace or special characters).

It also warns about `let` being undefined in POSIX shell, but `let` is defined in the shell script so it's a false positive that's caused by the use of the `let` keyword specifically.

If you think there are other issues or ways to improve Pnut's compatibility with Shellcheck, please let us know!

0: https://www.shellcheck.net/wiki/SC1102

From our experience, ksh is generally faster, and dash sits between ksh and bash. One reason is that dash stores variables using a very small hash table with only 37 entries[0] meaning variable access quickly becomes linear as memory usage grows. But even with that, dash is still surprisingly fast -- when compiling `pnut.c` with `pnut.sh`, dash comes in second place:

  ksh93: 31s
  dash:  1m06s
  bash:  1m19s
  zsh:   >15m

[0]: https://git.kernel.org/pub/scm/utils/dash/dash.git/tree/src/...

EDIT: ksh93, not ksh

One of the example we include is a base64 encoder/decoder:

  https://github.com/udem-dlteam/pnut/blob/main/examples/compiled/base64.sh

It doesn't support NULs as you pointed out, but it's interesting to see similarities between your implementation and the one generated by Pnut.

Because we use `read -r`, we haven't tested reading binary files. Fortunately, the shell's `printf` function can emit all 256 characters so Pnut can at least output binary files. This makes it possible for Pnut to have a x86 backend for the use of reproducible builds.

Regarding the use of `read`, one constraint we set ourselves when writing Pnut is to not use any external utilities, including those that are specified by the POSIX standard (other than `read` and `printf`). This maximizes portability of the code generated by Pnut and is enough for the reproducible build use case.

We're still looking for ways to integrate existing shell code with C. One way this can be done is through the use of the `#include_shell` directive which includes existing shell code in the generated shell script. This makes it possible to call the necessary utilities to read raw bytes without having Pnut itself depends on less portable utilities.

Author here,

That's correct! Unlike Bash and other modern shells, the POSIX standard doesn't include arrays or any other data structures. The way we found around this limitation is to use arithmetic expansion and indexed shell variables (that are starting with `_` as you noted) to get random memory access.

Author here,

Because all shell variables in code generated by pnut are numbers, variables never contain whitespace or special characters and don't need to be quoted. We considered quoting all variable expansions as this is generally seen as best practice in shell programming, but thought it hurt readability and decided not to.

If you think there are other issues, please let me know!