Re:Introducing Structured Outputs in the API
by BoorishBears on Tuesday August 06, 2024 @02:14PM
(#41173549)
I was surprised it took so long until I reached this line:
> The model can fail to follow the schema if the model chooses to refuse an unsafe request. If it chooses to refuse, the return message will have the refusal boolean set to true to indicate this.
I'm not sure how they implemented that, maybe they've figured out a way to give the grammar a token or set of tokens that are always valid mid generation and indicate the model would rather not continue generating.
Right now JSON generation is one of the most reliable ways to get around refusals, and they managed not to introduce that weakness into their model
> The model can fail to follow the schema if the model chooses to refuse an unsafe request. If it chooses to refuse, the return message will have the refusal boolean set to true to indicate this.
I'm not sure how they implemented that, maybe they've figured out a way to give the grammar a token or set of tokens that are always valid mid generation and indicate the model would rather not continue generating.
Right now JSON generation is one of the most reliable ways to get around refusals, and they managed not to introduce that weakness into their model