Re:A cartel of influential datasets are dominating machine l...
by throwawayay02 on Monday December 06, 2021 @07:11AM
(#29458754)
Do they really reject them though? In my experience publishers will publish anything, if anything they should have higher standards not lower.
Re:A cartel of influential datasets are dominating machine l...
by domenicrosati on Monday December 06, 2021 @07:24AM
(#29458858)
For brand name journals and conferences, yes they do. My experience with those is a very high rejection rate.
There is a good reason to reject if you are not using a standard dataset. How can you compare the results of two approaches, to say natural language inference of whether one sentence entails another, without results being tested on the same dataset?
I think one thing overlooked in the conversation is that many papers start with a standard baseline and then use another dataset to establish additional results.
In my experience in nlp also, journals and conferences tend to establish datasets of their own when they make a call for submission. Often these are called the "shared task" track. ACL has operated this way for decades.
The problem here is, though, most publishers don't accept papers if the new proposal isn't backed by benchmarks by the well-known datasets. Even though it can be a competitive approach for a specific field, they reject the paper anyway if it's not performed well on the datasets.