Re:Structured Output from LLMs (Without Reprompting!)
by brucethemoose2 on Sunday July 16, 2023 @11:05PM
(#36753611)
> Which brings me to the other approach: steering the LLM's output __as it is generating tokens__
A relevant PR:
https://github.com/ggerganov/llama.cpp/pull/1773
The plan is to support arbitrary grammar files to constrain tokens generation, similar to the grammar files here:
Re:Structured Output from LLMs (Without Reprompting!)
by kelseyfrog on Monday July 17, 2023 @10:51AM
(#36758970)
I mean even Jsonformer used LogitsWarper when generating numbers, but yes arbitrary grammars are infinitely more powerful.
Re:Structured Output from LLMs (Without Reprompting!)
by darkteflon on Monday July 17, 2023 @12:51AM
(#36754313)
I'd never heard of LMQL before today, but it looks very nice. Do you have any experience building with it and, if so, would you be willing to comment on what it's like?
Re:Structured Output from LLMs (Without Reprompting!)
by theblazehen on Monday July 17, 2023 @03:22AM
(#36755122)
I did a POC project with it recently. The guidance on gpt-3.5-turbo and gpt-4 models isn't as functional as plain gpt-3. I found I had better results using https://github.com/piercefreeman/gpt-json and it doesn't require multiple calls to the API. Not as feature filled, but it may meet your needs
Re:Structured Output from LLMs (Without Reprompting!)
by darkteflon on Monday July 17, 2023 @07:49AM
(#36756959)
Thanks for the recommendation - gpt-json looks quite nice, actually - will check it out.
Re:Structured Output from LLMs (Without Reprompting!)
by sebmellen on Monday July 17, 2023 @01:34AM
(#36754509)
Yes! I don’t have much to say that wouldn’t be restating what I’ve already written, so I’ll link this for reference: https://www.sebastianmellen.com/post/2023/the-killer-use-cas....
Re:Structured Output from LLMs (Without Reprompting!)
by lbeurerkellner on Monday July 17, 2023 @02:50AM
(#36754911)
Thank you for bringing up LMQL. We have active branches with regex, parsers and types (structured and other types) which will also soon be upstreamed, making typed LLM use more advanced than the current template-based approach we support.
Re:Structured Output from LLMs (Without Reprompting!)
by sandGorgon on Monday July 17, 2023 @03:10AM
(#36755037)
has there been any work done to finetune OSS models to behave the same as openai functions (to constrain json output) ?
this is tablestakes now, but it doesnt seem ANY opensource model has this capability
Re:Structured Output from LLMs (Without Reprompting!)
by sandkoan on Monday July 17, 2023 @03:16AM
(#36755083)
You wouldn't actually want to, because you'd be losing generalizability, and it's a lot of unnecessary work.
I think approach #1 outlined above is the better (more cost- and time-efficient) technique—where a pretrained model already understands JSON (among myriad other formats), and you merely constrain it at text-gen time to valid JSON (or other format).
Re:Structured Output from LLMs (Without Reprompting!)
by sandGorgon on Tuesday July 18, 2023 @03:18AM
(#36768977)
so im not sure what is the difference between what you wrote and what i wrote ? are you distinguishing between "pretrained models" (as in base models) and finetuned models ?
here's my question then - was the GPT 0613 update (which introduced functions) a completely new base model or simply a finetuned model ? it seems to be the latter.
Re:Structured Output from LLMs (Without Reprompting!)
by sandkoan on Tuesday July 18, 2023 @04:23AM
(#36769426)
Yeah, then it seems we agree. I was just pointing out that it's not necessary to finetune OSS models to behave like OpenAI functions if you're able to do something similar to what we did (no tuning involved!).
1. LMQL/guidance/JSONformer/OP's post
2. finetuning the model to understand function calls and their (potentially) JSON schemas.
there was a comment here about OpenAI's approach (finetuning a model to understand function call) which raised a good point: since finetuning is often forgetful (previous knowledge learnt by the model gets forgotten a little bit), it's not clear if OpenAI's approach has made GPT-4 less capable than it was before. Not to mention that you're still dealing with a statistical process (LLM), not a locked-in algorithm that generates the desired schema 100% the time.
Which brings me to the other approach: steering the LLM's output __as it is generating tokens__, which is what LMQL does. This results in less token usage (you don't send function schema as part of your prompt/message to OpenAI) and 100% accuracy because token probabilities are modified (e.g., 0% chance of any character except ":" after a double quotation mark).