News for hackers. Stuff that matters.
Re:Show HN: LLMs can generate valid JSON 100% of the time by burke on Monday August 14, 2023 @05:35PM (#37127101)
torch-grammar generates a mask per PDA stack... we don't try to compute all the possible stacks. I'm sure there's something smarter that could be done here and you've probably figured it out (though IIRC regular languages don't have the arbitrarily recursive stack problem that you get when you get to context-free languages?) anyway, in practice we spend a few milliseconds on the first few requests building caches and then just apply masks from caches after that.
Re:Show HN: LLMs can generate valid JSON 100% of the time by remilouf on Tuesday August 15, 2023 @10:48AM (#37134653)
Sorry for misrepresenting your work. Thank you for correcting me and the explanation. Will take a closer look.