Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin...
by og_kalu on Wednesday October 30, 2024 @10:29PM
(#42002850)
Table 1 in the Appendix. GSM-No-op is the one benchmark that sees significant drops for those 4 models as well (with preview dropping the least at -17%).
No-op adds "seemingly relevant but ultimately inconsequential statements". So "change names, performance drops" is decidedly false for today's state of the art.
[ Parent
| on HN ]
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin...
by gruez on Wednesday October 30, 2024 @10:39PM
(#42002919)
Thanks. I wrongly focused on the headline result of the paper rather than the specific claim in the comment chain about "changing name, different results".
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin...
by TaylorAlexander on Thursday October 31, 2024 @03:24AM
(#42004236)
Ah, that’s a good point thanks for the correction.