News for hackers. Stuff that matters.
Re:GPT4 Can’t Ace MIT by iudqnolq on Saturday June 17, 2023 @07:35PM (#36375853)
ImageNet has five orders of magnitude more answers, which I would assume makes QA a completely different category of problem.

The authors could probably have carefully review all ~300 of their questions. If they couldn't they could have just reduced their sample size to say 50.

Re:GPT4 Can’t Ace MIT by muds on Saturday June 17, 2023 @08:19PM (#36376109)
I admit that Imagenet isn't the best analogy here. But I'm pretty confident that this data cleaning issue would be caught in peer review. The biggest issue which I still don't understand was the removal of the test set. That was bad practice on the authors' part.
Re:GPT4 Can’t Ace MIT by iudqnolq on Saturday June 17, 2023 @09:02PM (#36376321)
In general with evaluations LLVMs I keep seeing issues that would be caught by a human carefully looking at the results. To a nonexpert who occasionally peeks at things it seems like having a small dataset that's just slightly too big for you to manually review is a bad practice.

It also seems like 100% accuracy should have raised red flags, especially if you know your dataset isn't perfectly cleaned.