News for hackers. Stuff that matters.
Re:Task-Specific LLM Evals That Do and Don't Work by behnamoh on Tuesday December 10, 2024 @10:22AM (#42377660)
OP here. I liked Macro-o1 (Marco1?) but as you pointed out, we need to teach these models to spend their system 2 thinking more economically.
Re:Task-Specific LLM Evals That Do and Don't Work by caturopath on Wednesday December 11, 2024 @08:09PM (#42395054)
One of the things complicating the example is that the counting-letters task is actually a legitimately hard one for it, even if it's trivial for us: it's bad at letter stuff because of the way it represents text with tokens. I believe the problem exists, but the letter thing isn't necessarily a great example of something that it should recognize as trivial.