News for hackers. Stuff that matters.
Re:Task-Specific LLM Evals That Do and Don't Work by qeternity on Monday December 09, 2024 @12:14PM (#42368045)
This is not true at all. Just because you can force the logits to give syntactically valid outputs, doesn't mean you're going to get a useful result.

Constrained generation, without a proper understanding of the model's natural response tendencies, can give horrible results.

Re:Task-Specific LLM Evals That Do and Don't Work by msp26 on Monday December 09, 2024 @01:07PM (#42368684)
I agree with you completely. I was talking about the parsing being easy with this, not referring to the outputs being correct in reality.

You can get awful results with poorly defined constraints.

Re:Task-Specific LLM Evals That Do and Don't Work by imtringued on Monday December 09, 2024 @01:40PM (#42369055)
Depends on the way you do constrained generation. If all you do is reject tokens using a grammar, then yeah it is bad. If your software inserts things like field names and braces instead of forcing the model to produce them token by token and then afterwards rejecting the wrong tokens, then you should be good to go.
Re:Task-Specific LLM Evals That Do and Don't Work by potatoman22 on Monday December 09, 2024 @11:34PM (#42373794)
Why is it bad to reject tokens using a grammar?