>That being said, I'm starting to doubt the leaderboards as an accurate representation of model ability

Goodhart's law applies here just like everywhere else. Much more so given how much money these companies are dumping into making these models.