Re:New Top Score on AI2 Reasoning Challenge (ARC) Is 53.84%
by yorwba on Saturday January 05, 2019 @02:29PM
(#18833665)
This leaderboard tries to prevent that kind of problem:
> The dataset is partitioned into a Challenge Set and an Easy Set, where the former contains only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurrence algorithm. This leaderboard is for the Challenge Set.
Additionally, I don't think they let you try often enough to get a meaningful chance at significantly beating the baseline with just pure randomness.
> The dataset is partitioned into a Challenge Set and an Easy Set, where the former contains only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurrence algorithm. This leaderboard is for the Challenge Set.
Additionally, I don't think they let you try often enough to get a meaningful chance at significantly beating the baseline with just pure randomness.