News for hackers. Stuff that matters.
Re:Inversion: Fast, Reliable Structured LLMs by zuck_vs_musk on Tuesday March 19, 2024 @02:49AM (#39753476)
> strong guarantees on the output (i.e. 100% reliable, without requiring any retries).

Has anyone seen a good JSON library that can handle slightly broken JSON? e.g. trailing commas, unescaped newlines, etc.? I have not found a good one.

Re:Inversion: Fast, Reliable Structured LLMs by reissbaker on Tuesday March 19, 2024 @03:21AM (#39753604)
Entirely broken JSON, no — I would be surprised if one existed. If you want slightly laxer semantics like trailing commas, JSON5 [1] is a pretty good spec and is JSON-compatible. I used to use it for LLMs (while telling them to emit JSON — no need to confuse them by explaining JSON5), in order to handle things like trailing commas, but in my experience LLMs have gotten good enough over the last year I mostly don't even bother anymore.

1: https://json5.org/

Re:Inversion: Fast, Reliable Structured LLMs by catlifeonmars on Tuesday March 19, 2024 @04:36AM (#39753882)
Should be easy to build on top of a lexer as a pre-parsing pass.
Re:Inversion: Fast, Reliable Structured LLMs by tlarkworthy on Tuesday March 19, 2024 @06:55AM (#39754501)
Dirtyjson
Re:Inversion: Fast, Reliable Structured LLMs by dartos on Tuesday March 19, 2024 @08:05AM (#39754889)
You can force LLMs to generate valid json by using a context free grammar FWIW
Re:Inversion: Fast, Reliable Structured LLMs by zuck_vs_musk on Tuesday March 19, 2024 @11:55PM (#39762734)
Please elaborate
Re:Inversion: Fast, Reliable Structured LLMs by kaiokendev on Wednesday March 20, 2024 @08:34AM (#39765545)
Matt Rickard has a good entry level blog post about it, from the angle of regex constraining [0]. Context free grammars follow the same principle, except using a finite state machine to restrict the action space.

[0]: https://matt-rickard.com/rellm

Re:Inversion: Fast, Reliable Structured LLMs by dartos on Thursday March 21, 2024 @05:34PM (#39784675)
Look at llama.cpp grammars, lmql, or guidance-ai