News for hackers. Stuff that matters.
Re:Structured Output from LLMs (Without Reprompting!) by brucethemoose2 on Sunday July 16, 2023 @11:05PM (#36753611)
> Which brings me to the other approach: steering the LLM's output __as it is generating tokens__

A relevant PR:

https://github.com/ggerganov/llama.cpp/pull/1773

The plan is to support arbitrary grammar files to constrain tokens generation, similar to the grammar files here:

https://github.com/antlr/grammars-v4

Re:Structured Output from LLMs (Without Reprompting!) by kelseyfrog on Monday July 17, 2023 @10:51AM (#36758970)
I mean even Jsonformer used LogitsWarper when generating numbers, but yes arbitrary grammars are infinitely more powerful.