Re:Structured Output from LLMs (Without Reprompting!)
by sandkoan on Sunday July 16, 2023 @09:58PM
(#36753254)
The prompt is given to our model as a guiding aid (a suggestion), and the cfg is used to constrain the model to generate only tokens that abide by the schema (an enforcement). That's how we ensure only valid outputs at text generation time.
We also prefill some tokens depending on the set of allowed tokens at a given state, so the model doesn't waste resources trying to predict them.
When you say in another comment that using OpenAI functions to output JSON is a waste of tokens, how are you generating the JSON output? And why do your prompts then include few shot examples of JSON objects?