News for hackers. Stuff that matters.
Re:NixOS Reproducible Builds: minimal ISO successfully indep... by drdrey on Sunday October 29, 2023 @12:48PM (#38060340)
a very common one is pointer values being different from run to run and across different operating systems. Any code that intentionally or accidentally relies on pointer values will be non-deterministic
Re:NixOS Reproducible Builds: minimal ISO successfully indep... by dataflow on Sunday October 29, 2023 @12:52PM (#38060379)
Would be nice if you could explain how/why this happens, given that normally, pointers aren't persisted.
Re:NixOS Reproducible Builds: minimal ISO successfully indep... by traxys on Sunday October 29, 2023 @12:57PM (#38060430)
I think they meant if you cast a pointer to an integer, do some math on that and then store that. Then you will a stored result that will likely differ from run to run
Re:NixOS Reproducible Builds: minimal ISO successfully indep... by edgyquant on Sunday October 29, 2023 @01:06PM (#38060526)
That sounds like runtime differences not a difference between two binaries
Re:NixOS Reproducible Builds: minimal ISO successfully indep... by stcg on Sunday October 29, 2023 @01:50PM (#38060944)
The difference in binaries must be caused by some runtime difference of a compiler.
Re:NixOS Reproducible Builds: minimal ISO successfully indep... by drdrey on Tuesday October 31, 2023 @07:07PM (#38092591)
That's right, look at this thread for example:

https://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20...

The Global Value Numbering pass in LLVM was iterating over `DenseMap<BasicBlock*, ...>`, so the iteration order was dependent on the value of BasicBlock pointers. This could lead to the same source files and compiler producing different binaries.

Re:NixOS Reproducible Builds: minimal ISO successfully indep... by someplaceguy on Sunday October 29, 2023 @02:03PM (#38061045)
Languages such as Standard ML and others (Scheme? Lisp? Not sure...) have implementations that can save the current state of the heap into a binary.

This is used in theorem provers, for example, so that you don't have to verify proofs of theorems over and over again (which can be very slow).

Instead, you verify them once, save the state of the heap to disk (as a binary ELF, for instance) and then you can run the binary to continue exactly where you left off (i.e. with all the interesting theorems already in memory, in a proved state).

This is what the HOL4 theorem prover's main `hol` script does, i.e. it runs HOL4 by loading such a memory state from disk, with the core theories and theorems already loaded.

Presumably, to make this reproducible you'd need to make sure that all the memory objects are saved to disk in a deterministic order somehow (e.g. not in memory address order, as it can change from run to run, especially with ASLR and/or when using multiple threads).

Edit: Presumably you'd also need to make sure that you persist the heap when all threads are idle and in a known state (e.g. with all timers stopped), to avoid random stack states and extraneous temporary allocations from being persisted, which would also affect the resulting binary.

Re:NixOS Reproducible Builds: minimal ISO successfully indep... by dataflow on Sunday October 29, 2023 @02:14PM (#38061126)
Thanks, yeah. So I guess the concrete example I would cite here is that the most natural (and most efficient?) way of persisting std::map<ptr, ....> would introduce pointer ordering into the output.
Re:NixOS Reproducible Builds: minimal ISO successfully indep... by someplaceguy on Sunday October 29, 2023 @02:58PM (#38061518)
Just like the most natural (and most efficient?) way of persisting any std::unordered_map<...> can result in a completely randomly-ordered output, due to a DoS mitigation that some commonly-used language runtimes have.
Re:NixOS Reproducible Builds: minimal ISO successfully indep... by edgyquant on Sunday October 29, 2023 @01:06PM (#38060534)
That’s runtime behavior