News for hackers. Stuff that matters.
Re:Introducing Structured Outputs in the API by visarga on Tuesday August 06, 2024 @05:00PM (#41175331)
> The same input will always produce the same output

Not guaranteed even with the same seed. If you don't perform all operations in exactly the same order, even a simple float32 sum, if batched differently, will result in different final value. This depends on the load factor and how resources are allocated.

Re:Introducing Structured Outputs in the API by simonw on Tuesday August 06, 2024 @05:03PM (#41175360)
Yeah, the fact that floating point multiplication isn't associative is a real pain for producing deterministic outputs - especially when you're running massively parallel computations on GPUs (or multiple GPUs) making the order of operations even less predictable.
Re:Introducing Structured Outputs in the API by taneq on Wednesday August 07, 2024 @01:39AM (#41178528)
This doesn’t mean LLMs are inherently non-deterministic, just that current common implementations are non-deterministic.