News for hackers. Stuff that matters.
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by og_kalu on Wednesday October 30, 2024 @10:29PM (#42002850)
Table 1 in the Appendix. GSM-No-op is the one benchmark that sees significant drops for those 4 models as well (with preview dropping the least at -17%). No-op adds "seemingly relevant but ultimately inconsequential statements". So "change names, performance drops" is decidedly false for today's state of the art.
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by gruez on Wednesday October 30, 2024 @10:39PM (#42002919)
Thanks. I wrongly focused on the headline result of the paper rather than the specific claim in the comment chain about "changing name, different results".
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin... by TaylorAlexander on Thursday October 31, 2024 @03:24AM (#42004236)
Ah, that’s a good point thanks for the correction.