There is a good reason to reject if you are not using a standard dataset. How can you compare the results of two approaches, to say natural language inference of whether one sentence entails another, without results being tested on the same dataset?

I think one thing overlooked in the conversation is that many papers start with a standard baseline and then use another dataset to establish additional results.

In my experience in nlp also, journals and conferences tend to establish datasets of their own when they make a call for submission. Often these are called the "shared task" track. ACL has operated this way for decades.