Re:Inversion: Fast, Reliable Structured LLMs
by dwallin on Tuesday March 19, 2024 @08:49AM
(#39755167)
I thought they pretty clearly explained where the additional performance came from. If there is only 1 valid schema confirming option you can skip the LLM entirely. If there are only a limited number of possible tokens (eg. only } or ,) then you run it on smaller subset of the model. Between these two you capture a large amount of the actual token count of most json.
[ Parent
| on HN ]
There are no comments on this comment yet.