News for hackers. Stuff that matters.
Re:Task-Specific LLM Evals That Do and Don't Work by jacobr1 on Monday December 09, 2024 @07:34PM (#42372283)
Here is a short example that came up for me last week.

I had a set of documents I wanted to classify according a taxonomy that is well known (so it is exists in the training data of all the major llm models I tested)

If I have prompt like, `You are an expert classification system. Using the Classification Approach Foo, consider the following and output the category in JSON format, such as {"class":"bar"} `

This works ok, but it works much better if I tell it to output {"class":"bar", "reason": "baz"} and improved with some other approaches like adding "related_class" or "parent_category" which would otherwise be redundant.

Also including some few-shot examples helped, but the biggest benefit came from the "reason" field. Trying justification or other synonyms seems to produce the same output.

I suspect this is something similar to CoT.

Re:Task-Specific LLM Evals That Do and Don't Work by glaugh on Monday December 09, 2024 @09:28PM (#42373085)
Great, thank you (and hedgehog), that makes sense
Re:Task-Specific LLM Evals That Do and Don't Work by potatoman22 on Monday December 09, 2024 @11:33PM (#42373788)
Have you tested moving the "reason" field before the "class" field? That may encourage better CoT instead of having the model justify the class after it already picked it. Anecdotally, I saw a 5% boost in performance from a NER system from having the model output the entity's class at the end rather than the beginning.
Re:Task-Specific LLM Evals That Do and Don't Work by te_chris on Tuesday December 10, 2024 @04:30AM (#42375235)
This has worked for me for sentiment classification, i had a score and a summary field and the results were much more accurate.
Re:Task-Specific LLM Evals That Do and Don't Work by andai on Tuesday December 10, 2024 @10:05AM (#42377521)
Does it still work if the class field is output before the reason?
Re:Task-Specific LLM Evals That Do and Don't Work by jacobr1 on Tuesday December 10, 2024 @12:06PM (#42378732)
Yes. But, reason first did work (very slightly) better.