Re:GPT4 Can’t Ace MIT
by muds on Saturday June 17, 2023 @08:19PM
(#36376109)
I admit that Imagenet isn't the best analogy here. But I'm pretty confident that this data cleaning issue would be caught in peer review. The biggest issue which I still don't understand was the removal of the test set. That was bad practice on the authors' part.
[ Parent
| on HN ]
In general with evaluations LLVMs I keep seeing issues that would be caught by a human carefully looking at the results. To a nonexpert who occasionally peeks at things it seems like having a small dataset that's just slightly too big for you to manually review is a bad practice.
It also seems like 100% accuracy should have raised red flags, especially if you know your dataset isn't perfectly cleaned.