News for hackers. Stuff that matters.
Re:The Coming Knowledge-Work Supply-Chain Crisis by giantrobot on Sunday April 27, 2025 @03:17PM (#43814387)
> That they can be randomly, nondeterministically and confidently wrong, and there is no way to know without manually reviewing every output.

This is my exact same issue with LLMs and it's routinely ignored by LLM evangelists/hypesters. It's not necessarily about being wrong it's the non-deterministic nature of the errors. They're not only non-deterministic but unevenly distributed. So you can't predict errors and need expertise to review all the generated content looking for errors.

There's also not necessarily an obvious mapping between input tokens and an output since the output depends on the whole context window. An LLM might never tell you to put glue on pizza because your context window has some set of tokens that will exclude that output while it will tell me to do so because my context window doesn't. So there's not even necessarily determinism or consistency between sessions/users.

I understand the existence of Gell-Mann amnesia so when I see an LLM give confident but subtly wrong answers about a Python library I don't then assume I won't also get confident yet subtly wrong answers about the Parisian Metro or elephants.

Re:The Coming Knowledge-Work Supply-Chain Crisis by furyofantares on Sunday April 27, 2025 @04:01PM (#43814668)
This is a nitpick because I think your complaints are all totally valid, except that I think blaming non-determinism isn't quite right. The models are in fact deterministic. But that's just technical, from a practical sense they are non-deterministic in that a human can't determine what it'll produce without running it, and even then it can be sensitive to changes in context window like you said, so even after running it once you don't know you'll get a similar output from similar inputs.

I only post this because I find it kind of interesting; I balked at blaming non-determinism because it technically isn't, but came to conclude that practically speaking that's the right thing to blame, although maybe there's a better word that I don't know.

Re:The Coming Knowledge-Work Supply-Chain Crisis by ryandrake on Sunday April 27, 2025 @04:18PM (#43814780)
> from a practical sense they are non-deterministic in that a human can't determine what it'll produce without running it

But this is also true for programs that are deliberately random. If you program a computer to output a list of random (not pseudo-random) numbers between 0 and 100, then you cannot determine ahead of time what the output will be.

The difference is, you at least know the range of values that it will give you and the distribution, and if programmed correctly, the random number generator will consistency give you numbers in that range with the expected probability distribution.

In contrast, an LLM's answer to "List random numbers between 0 and 100" usually will result in what you expect, or (with a nonzero probability) it might just up and decide to include numbers outside of that range, or (with a nonzero probability) it might decide to list animals instead of numbers. There's no way to know for sure, and you can't prove from the code that it won't happen.

Re:The Coming Knowledge-Work Supply-Chain Crisis by ijk on Monday April 28, 2025 @10:58AM (#43822217)
> it might just up and decide to include numbers outside of that range, or (with a nonzero probability) it might decide to list animals instead of numbers

For example, all of the replies I've gotten that are formatted as "Here is the random number you asked for: forty-two."

Which is both absolutely technically correct and very completely missing the point, and it might decide to do that one time in a hundred and crash your whole stack.

There are ways around that, but it's a headache you don't get with rand() or the equivalent for whatever problem you're solving.

Re:The Coming Knowledge-Work Supply-Chain Crisis by giantrobot on Sunday April 27, 2025 @05:13PM (#43815178)
At the base levels LLMs aren't actually deterministic because the model weights are typically floats of limited precision. At a large enough scale (enough parameters, model size, etc) you will run into rounding issues that effectively behave randomly and alter output.

Even with temperature of zero floating point rounding, probability ties, MoE routing, and other factors make outputs not fully deterministic even between multiple runs with identical contexts/prompts.

In theory you could construct a fully deterministic LLM but I don't think any are deployed in practice. Because there's so many places where behavior is effectively non-deterministic the system itself can't be thought of as deterministic.

Errors might be completely innocuous like one token substituted for another with the same semantic meaning. An error might also completely change the semantic meaning of the output with only a single token change like an "un-" prefix added to a word.

The non-determinism is both technically and practically true in practice.

Re:The Coming Knowledge-Work Supply-Chain Crisis by mrob on Sunday April 27, 2025 @08:01PM (#43816222)
Most floating point implementations have deterministic rounding. The popular LLM inference engine llama.cpp is deterministic when using the same sampler seed, hardware, and cache configuration.
Re:The Coming Knowledge-Work Supply-Chain Crisis by Earw0rm on Monday April 28, 2025 @03:17AM (#43818514)
Non-explicable?

It's deterministic in that (input A, state B) always produces output C. But it can't generally be reasoned about, in terms of how much change to A will produce C+1, nor can you directly apply mechanical reasoning to /why/ (A.B) produces C and get a meaningful answer.

(Yes, I know, "the inputs multiplied by the weights", but I'm talking about what /meaning/ someone might ascribe to certain weights being valued X, Y or Z in the same sense as you'd look at a variable in a running program or a physical property of a mechanical system).

Re:The Coming Knowledge-Work Supply-Chain Crisis by gopher_space on Sunday April 27, 2025 @04:01PM (#43814665)
The prompts we’re using seem like they’d generate the same forced confidence from a junior. If everything’s a top-down order, and your personal identity is on the line if I’m not “happy” with the results, then you’re going to tell me what I want to hear.
Re:The Coming Knowledge-Work Supply-Chain Crisis by giantrobot on Sunday April 27, 2025 @05:26PM (#43815285)
There's some differences between junior developers and LLMs that are important. For one a human developer can likely learn from a mistake and internalize a correction. They might make the mistake once or twice but the occurrences will decrease as they get experience and feedback.

LLMs as currently deployed don't do the same. They'll happily make the same mistake consistently if a mistake is popular in the training corpus. You need to waste context space telling them to avoid the error until/unless the model is updated.

It's entirely possible for good mentors to make junior developers (or any junior position) feel comfortable being realistic in their confidence levels for an answer. It's ok for a junior person to admit they don't know an answer. A mentor requiring a mentee to know everything and never admit fault or ignorance is a bad mentor. That's encouraging thought terminating behavior and helps neither person.

It's much more difficult to alter system prompts or get LLMs to even admit when they're stumped. They don't have meaningful ways to even gauge their own confidence in their output. Their weights are based on occurrences in training data rather than correctness of the training data. Even with RL the weight adjustments are only as good as the determinism of the output for the input which is not great for several reasons.

Re:The Coming Knowledge-Work Supply-Chain Crisis by alabastervlog on Monday April 28, 2025 @08:58AM (#43821030)
The other day, Google's dumbshit search LLM thingy invented a command line switch that doesn't exist, told me how it works, and even provided warnings for common pitfalls.

For something it made up.

That's a bit more than an embarrassed junior will do to try to save face, usually.