News for hackers. Stuff that matters.
Re:Inversion: Fast, Reliable Structured LLMs by dwallin on Tuesday March 19, 2024 @08:49AM (#39755167)
I thought they pretty clearly explained where the additional performance came from. If there is only 1 valid schema confirming option you can skip the LLM entirely. If there are only a limited number of possible tokens (eg. only } or ,) then you run it on smaller subset of the model. Between these two you capture a large amount of the actual token count of most json.

There are no comments on this comment yet.