News for hackers. Stuff that matters.
Re:Introducing Structured Outputs in the API by thruway516 on Tuesday August 06, 2024 @03:32PM (#41174385)
I dont understand your complaint at all. If you develop a new revolutionary technology called an automobile, developing steering, brakes, starter, mufflers for it is a pretty big deal even if reins, clamps, mufflers and keys are mundane and have existed for decades. Structured outputs are a pretty big step in making this magic actually usable by developers as opposed to generating impressive cat pictures or whatever has captured the public imagination.
Re:Introducing Structured Outputs in the API by Bjartr on Tuesday August 06, 2024 @03:36PM (#41174421)
I don't think it was an complaint, just a observation.
Re:Introducing Structured Outputs in the API by thruway516 on Tuesday August 06, 2024 @04:21PM (#41174912)
Yes probably. But considering non-deterministic outputs is the nature of the beast with Llms and we're (mostly) engineers here, calling any part of this mundane sounds almost more like fighting words than just observation
Re:Introducing Structured Outputs in the API by the8thbit on Tuesday August 06, 2024 @04:51PM (#41175218)
Extremely pedantic, but is "non-deterministic" really the right language? The same input will always produce the same output, provided you haven't intentionally configured the system to use the model non-deterministically. It seems like the right way to describe it is as a chaotic deterministic system. The same input will always produce the same output, but small shifts in the input or weights can result in dramatic and difficult to predict changes in outputs.
Re:Introducing Structured Outputs in the API by visarga on Tuesday August 06, 2024 @05:00PM (#41175331)
> The same input will always produce the same output

Not guaranteed even with the same seed. If you don't perform all operations in exactly the same order, even a simple float32 sum, if batched differently, will result in different final value. This depends on the load factor and how resources are allocated.

Re:Introducing Structured Outputs in the API by simonw on Tuesday August 06, 2024 @05:03PM (#41175360)
Yeah, the fact that floating point multiplication isn't associative is a real pain for producing deterministic outputs - especially when you're running massively parallel computations on GPUs (or multiple GPUs) making the order of operations even less predictable.
Re:Introducing Structured Outputs in the API by taneq on Wednesday August 07, 2024 @01:39AM (#41178528)
This doesn’t mean LLMs are inherently non-deterministic, just that current common implementations are non-deterministic.
Re:Introducing Structured Outputs in the API by davedx on Tuesday August 06, 2024 @05:00PM (#41175321)
Llms are indeed non deterministic
Re:Introducing Structured Outputs in the API by jappgar on Tuesday August 06, 2024 @05:46PM (#41175720)
Structured outputs are hard... but they claimed to have solved this a year ago.

They were lying, of course, and meanwhile charged output tokens for malformed JSON.

Re:Introducing Structured Outputs in the API by jiggawatts on Tuesday August 06, 2024 @10:30PM (#41177694)
Structured output is trivial: just select the output tokens from the given list of probability values filtered for the allowed next tokens in the schema.

Other LLM vendors figured this out many months ago.

Re:Introducing Structured Outputs in the API by retinaros on Wednesday August 07, 2024 @06:27PM (#41186218)
they did not . anthropic often sends broken json