Re:Introducing Structured Outputs in the API
by jiggawatts on Tuesday August 06, 2024 @10:30PM
(#41177694)
Structured output is trivial: just select the output tokens from the given list of probability values filtered for the allowed next tokens in the schema.
Other LLM vendors figured this out many months ago.
They were lying, of course, and meanwhile charged output tokens for malformed JSON.