News for hackers. Stuff that matters.
Re:LLMs as the new high level language by orbital-decay on Saturday February 07, 2026 @10:20PM (#46931025)
Reproducibility across all existing build systems took a decade of work involving everything from compilers to sandboxing, and a hard reproducibility guarantee in completely arbitrary cases is either impossible or needs deterministic emulators which are terribly slow. (e.g. builds that depend on hardware probing or a simulation result)

Input-to-output reproducibility in LLMs (assuming the same model snapshot) is a matter of optimizing the inference for it and fixing the seed, which is vastly simpler. Google for example serves their models in an "almost" reproducible way, with the difference between runs most likely attributed to batching.

Re:LLMs as the new high level language by sarchertech on Saturday February 07, 2026 @11:58PM (#46931427)
It’s not just about non-determinism, but about how chaotic LLMs are. A one word difference in a spec can and frequently does produce unrecognizably different output.

If you are using an LLM as a high level language, that means that every time you make a slight change to anything and “recompile” all of the thousands upon thousands of unspecified implementation details are free to change.

You could try to ameliorate this by training LLMs to favor making fewer changes, but that would likely end up encoding every bad architecture decisions made along the way and essentially forcing a convergence on bad design.

Fixing this I think requires judgment on a level far beyond what LLMs have currently demonstrated.

Re:LLMs as the new high level language by orbital-decay on Sunday February 08, 2026 @01:37AM (#46931864)
>It’s not just about non-determinism

I'm very specifically addressing prompt reproducibility mentioned above, because it's a notorious red herring in these discussions. What you want is correctness, not determinism/reproducibility which is relatively trivial. (although thinking of it more, maybe not that trivial... if you want usable repro in the long run, you'll have to store the model snapshot, the inference code, and make it deterministic too)

>A one word difference in a spec can and frequently does produce unrecognizably different output.

This is well out of scope for the reproducibility and doesn't affect it in the slightest. And for practical software development this is also a red herring, the real issue is correctness and spec gaming. As long as the output is correct and doesn't circumvent the intention of the spec, prompt instability is unimportant, it's just the ambiguous nature of the domain LLMs and humans operate in.

Re:LLMs as the new high level language by sarchertech on Sunday February 08, 2026 @07:12AM (#46933604)
Well if you want to use it as a high level language where you check in the spec and regenerate the code then prompt instability/chaotic output makes that infeasible.

You can’t just tell users “sorry there are a million tiny differences all over the app every time we change the slightest thing, that’s just the ambiguous nature of reality”.

Re:LLMs as the new high level language by orbital-decay on Sunday February 08, 2026 @08:45AM (#46934142)
>where you check in the spec and regenerate the code then prompt instability/chaotic output makes that infeasible

What, why would you want to write the code anew? Identify the changes in the spec and bring the existing code in line with them.

Re:LLMs as the new high level language by sarchertech on Sunday February 08, 2026 @05:53PM (#46939423)
That’s the whole thesis of the article. Using an LLM as a high level language.

>The codebase should be reconstructable from the documentation

Re:LLMs as the new high level language by orbital-decay on Monday February 09, 2026 @01:36AM (#46942220)
And this kind of reproducibility is insensitive to prompt instability. If your inference is deterministic (trivial if within the lifetime of a snapshot), you give it the same docs and get the same result.
Re:LLMs as the new high level language by sarchertech on Monday February 09, 2026 @08:07AM (#46944822)
Obviously this article is advocating for spec as source code and not checking in code to keep it around between “compilations”.

Why would you care about reproducibility at all if you were checking in the code as well. If you check in the code, you’d never need to rerun the same prompt.

Re:LLMs as the new high level language by isodev on Sunday February 08, 2026 @07:28PM (#46940099)
You’re mixing the terms I think.

Reproducible builds is not the same as reproducible outcomes.

Rebuilding your project with a different version of a dependency is not the same as suddenly a program accepts txt files as attachments instead of pdfs because the model was borrowing a txt example from its training data and got sidetracked.