News for hackers. Stuff that matters.
Re:Structured Outputs in LLMs by 2THFairy on Tuesday September 23, 2025 @12:15PM (#45349070)
You don't need a new model. The trick of the technique is that you only change how tokens are sampled; Zero out the probability of every token that would be illegal under the grammar or other constraints.

All you need for that is an inference API that gives you the full output vector, which is trivial for any model you run on your own hardware.

There are no comments on this comment yet.