I suppose the situation regarding common datasets might vary between subfields and NLP tasks, so maybe I just saw a weird corner of it.
Of course the code was also nowhere to be seen.
Availability of code would of course be even more important, both because of replicability and general verifiability, and also because that would allow you to do a comparison with any number of datasets yourself.
Glad to hear that code availability has been improving.
> There are still many other problems, for example a big one is reporting of spurious improvements that can vanish if you get a less lucky random seed.
Considering that a lot of NLP is at least somewhat based on machine learning, don't people do cross-validation or something?
You do a paper showing that problem X can be solved slightly better by downloading and training on a billion tweets.
But you don’t have the copyright to those tweets, so you can’t share data.
> don't people do cross-validation or something
A lot of stable problems comes with a dataset already split into train and test.
That's true. Sometimes you might try to tweak the algorithm itself rather than the data, though, or experiment with different kinds of preprocessing or something, and in those cases it would be helpful to be able to do different experiments with shared datasets.
My limited experiences were from around the time deep learning was only about to become a big thing, so it might have been different then. Maybe you nowadays just throw more tweets and GPUs at the problem.
I don't have data, but from subjective experience, 5-6 years ago most papers in major NLP conferences didn't have an associated code repository. Now, the overwhelming majority do.
There are still many other problems, for example a big one is reporting of spurious improvements that can vanish if you get a less lucky random seed. But at least including code is now common practice.