News for hackers. Stuff that matters.
Re:Structured Output from LLMs (Without Reprompting!) by sandkoan on Sunday July 16, 2023 @09:58PM (#36753254)
The prompt is given to our model as a guiding aid (a suggestion), and the cfg is used to constrain the model to generate only tokens that abide by the schema (an enforcement). That's how we ensure only valid outputs at text generation time.

We also prefill some tokens depending on the set of allowed tokens at a given state, so the model doesn't waste resources trying to predict them.

Re:Structured Output from LLMs (Without Reprompting!) by roseway4 on Sunday July 16, 2023 @10:31PM (#36753423)
When you say “our model”, are you using a custom LLM for completions vs OpenAI or other LLM vendor?
Re:Structured Output from LLMs (Without Reprompting!) by sandkoan on Sunday July 16, 2023 @10:54PM (#36753547)
Custom LLM—hence the self-hostability.