Re:Show HN: LLMs can generate valid JSON 100% of the time
by remilouf on Monday August 14, 2023 @03:52PM
(#37125974)
Thanks for bringing clownfish and relm to my attention! afaik other libraries loop over the entire vocabulary at every step of the generation. We on the other hand build an index at initialization by looping once over the vocabulary. Then generation is just as fast as standard generation.
[ Parent
| on HN ]
Re:Show HN: LLMs can generate valid JSON 100% of the time
by burke on Monday August 14, 2023 @05:35PM
(#37127101)
torch-grammar generates a mask per PDA stack... we don't try to compute all the possible stacks. I'm sure there's something smarter that could be done here and you've probably figured it out (though IIRC regular languages don't have the arbitrarily recursive stack problem that you get when you get to context-free languages?) anyway, in practice we spend a few milliseconds on the first few requests building caches and then just apply masks from caches after that.