It also potentially gives every paper N replication problems to solve, in addition to just the gpu time. I would have to figure out HOW to retrain all of these models on the current form of the dataset... Which is fine for an occasional explicit replication study, but terrible if everyone has to do it.

I think it's probably better to have a (say) yearly release of the dataset, with results of some benchmark models released alongside the new version.

This is similar to how Common Voice is handling the problem: it's a crowd sourced, constantly growing dataset, which is awesome if you want to train in as much as possible for production models. You can get the whole current version any time, but they also have releases with a static fileset and train/test split, which should be better for research.