Re:Task-Specific LLM Evals That Do and Don't Work
by ActionHank on Monday December 09, 2024 @02:25PM
(#42369538)
I also firmly believe that number of tokens served is a metric that is tracked and encouraged to go up, because more tokens mean more charges. o1 "does more" by using a whole lot more tokens for a very slight bump in usefulness.
[ Parent
| on HN ]
Re:Task-Specific LLM Evals That Do and Don't Work
by baq on Tuesday December 10, 2024 @01:35AM
(#42374369)
The problem with such takes is 1) the facts are also consistent with performance plateauing for fundamental reasons and 2) there are good, perhaps better than them, competitors with similar issues.
Re:Task-Specific LLM Evals That Do and Don't Work
by ActionHank on Tuesday December 10, 2024 @05:41AM
(#42375605)
I think we’re on the same page here. In lieu of growing the product because of fundamental constraints they’ve opted to grow business metrics to net more revenue which in turn looks good on paper to investors.