News for hackers. Stuff that matters.
Re:Native JSON Output from GPT-4 by sanxiyn on Wednesday June 14, 2023 @03:42PM (#36331479)
Note that this (token selection restriction) is even available on OpenAI API as logit_bias.
Re:Native JSON Output from GPT-4 by newhouseb on Wednesday June 14, 2023 @04:45PM (#36332371)
But only for the whole generation. So if you want to constrain things one token at a time (as you would to force things to follow a grammar) you have to make fresh calls and only request one token which makes things more or less impractical if you want true guarantees. A few months ago I built this anyway to suss out how much more expensive it was [1]

[1] https://github.com/newhouseb/clownfish#so-how-do-i-use-this-...