Introducing Structured Outputs in the API

Re:Introducing Structured Outputs in the API by tcdent on Tuesday August 06, 2024 @02:08PM (#41173492)

GPT is still a language model, so at some point it's still just tokens.

Is this just a schema validation layer on their end to avoid the round trip (and cost) of repeating the call?

[ Parent | on HN ]

Re:Introducing Structured Outputs in the API by gamegoblin on Tuesday August 06, 2024 @02:12PM (#41173533)

Language models like GPT output a large vector of probabilities for the next token. Then a sampler decides which of those tokens to pick.

The simplest algorithm for getting good quality output is to just always pick the highest probability token.

If you want more creativity, maybe you pick randomly among the top 5 highest probability tokens or something. There are a lot of methods.

All that grammar-constrained decoding does is zero out the probability of any token that would violate the grammar.

[ Parent | on HN ]

Re:Introducing Structured Outputs in the API by nickreese on Tuesday August 06, 2024 @04:13PM (#41174834)

Thank you for this explanation. A few things just clicked for me.

[ Parent | on HN ]