News for hackers. Stuff that matters.
Re:Task-Specific LLM Evals That Do and Don't Work by petesergeant on Monday December 09, 2024 @11:11AM (#42367505)
This isn’t a problem in practice. Most of my prompts ask the LLM to do a bunch of chain of thought before asking them to spit out JSON. I extract the JSON, which works 97.5% of the time, and have a retry step being real specific about “here’s the conversation so far but I need JSON now” that handles the rest. Adding examples really helps.
Re:Task-Specific LLM Evals That Do and Don't Work by imtringued on Monday December 09, 2024 @01:43PM (#42369098)
https://lmsys.org/blog/2024-02-05-compressed-fsm/

I'm not trying to shill sglang specifically, just pointing out that there's a better way, btw.

Re:Task-Specific LLM Evals That Do and Don't Work by hansvm on Monday December 09, 2024 @02:39PM (#42369686)
...with the obvious caveat that the distribution of responses isn't the same

Elaborating slightly, retrying till the schema is adhered to has a different distribution from greedily selecting tokens adhering to the schema.

The simplest toy example I can come up with for that property is a universe of answers "aa", "ab", "bc", all of which the model is equally likely to output for a given prompt with normal auto-regressive invocations. The schema, in regex, is ".[bc]". Retry-till-success produces "ab" 1/2 of the time and "bc" the other half. Greedily adhering to the schema produces "ab" 2/3 of the time and "bc" the remaining third.

Last I checked large-scale LLMs, it was a problem in the wild for large string fields. They tend to want to finish the string with ellipses (this creating an incorrect response), but when they made that mistake they'd tend to truncate the entire json record and generate something that doesn't adhere to the schema. Retry-till-success has a high successful parse rate. Greedily adhering to the schema converts those ellipses errors into syntactically correct garbage.

Other such bugs can be much harder to quantify (model explainability is hard), but I'd be cautious employing the technique without a lot of case studies for your particular problem domain.