I actually disagree, but maybe not for the reasons you're expecting. I actually disagree because the reviews are unreasonably reasonable. They are void of context.

It's tough to explain, but I think it's also something every person who has written research papers can understand. How there's a big bias between reading and writing and how our works are not written to communicate our works as best as possible, but in effect how to communicate to reviewers that they should accept our works. The subtle distinction is deceptively large and I think we all could be a bit more honest about the system. After all, we want to optimize it, right?

The problem I see is all a matter of context. Good ideas often appear trivial once we see them. Often we fool ourselves into thinking that we already knew this, but do not have meaningful evidence that this is true but may try to reason that x = y + z + epsilon, but almost any idea can be framed that way, even breakthroughs like Evolution, Quantum Mechanics, or Relativity. It is because we look back at giants from a distance but when looking at works now don't see giants, but a bunch of children standing on one another's shoulders standing in a trench coat. That is the reality of it all. That few works are giant leaps and bounds, but rather incremental. The ones that take the biggest leaps are rare, often rely on luck (ambiguous definition), and frequently (but not always, especially considering the former) take a lot of time. Something we certainly don't often have.

We're trained as scientists and that means to be trained in critiquing systems and letting questions spiral. Sometimes the spiraling of questions shows how absurd an idea is but other times it can show how ingenious it is. It's easier to recognize the former but often hard to distinguish the latter. It is always easy to ask for more datasets, more experiments, and such, but these are critiques that can apply to any work as no work is complete. This is especially true in cases of larger breakthroughs, because any paradigm shift (even small) will cause a cascade of questions and create a lot of curiosity. But as we've all written papers, we know that this can often be a never ending cycle and often is quite impractical. The complaint about Table 4 is a perfect example. It is quite a difficult situation. The complaint is perfectly reasonable in that the question and concerns are quite valid and do warrant further understanding. But at the same time they are unreasonable because the requisite work required to answer these is not appropriate for the timescale that we work on. Do you have the compute or time to retrain all prior works to your settings? To retrain all your works to their settings? Maybe it doesn't work there which may or may not be evidence that the other works are just as good or not. What it comes down to is asking if these questions being answered could be another work in their own right. I'm not in NLP as deep as I'm in CV, but I suspect that the answer is yes (as in there are works that have been published answering exactly those questions).

There are also reasonably unreasonable questions in other respects. Such as the question about cosine distance vs Euclidean. This is one that I see quite often as we rely too deeply on our understanding of lower dimensional geometries to influence our understanding of high dimensional geometries. Such things that seem obvious, like distance, are quite ambiguous there and our example is the specific reason for the curse of dimensionality (it becomes difficult to distinguish the furthest point from the nearest point). But this often leads us in quite the wrong direction. Honestly, it is a bit surprising that the cosine similarity works (as D->inf cos(x,y)-> 0 forall x,y in R^D because any random vector is expected to be orthogonal meaning that to get cos(x,y)=1 means y = x + epsilon with epsilon -> 0 as D->inf. But I digress), but it does. There definitely could be entire works exploring these geometries and determining different geodesics. It is entirely enough for a work to simply have something working, even if it doesn't quite yet make sense.

The thing is that science is exceptionally fuzzy. Research is people walking around in the dark and papers are them communicating what we have found. I think it is important for us to remember this framing because we should then characterize the viability/publishability of a work not as illuminating everything but if the things found are useful (which itself is not always known). Because you might uncover a cavern and then it becomes easy to say "report back when you've explored it", but such an effort may be impossible to do alone. It can be a dead end, one that can take decades to explore (we'll always learn something though) or it may lead to riches we've never seen before. We don't know, but that's really what we're all looking for (hopefully more about riches for humanity than one's self, but one should be rewarded for sure).

This is why honestly, I advocate for abandoning the journal/conference system and leverage our modern tools like OpenReview to accelerate communication. Because it enables us to be more open about our limitations, to discuss our failures, and write to our peers rather than our critics. Critics are of course important, but they can take over too easily because they are reasonable and correct, but oft missing context. For an example, see the many HN comments about a technology in its infancy where people will complain that it is not yet competitive with existing technologies and thus devalue the potential. Oft missing the context that it takes time to compete, the time and energy put in by so many before us to make what we have now, but also that there are limits and despite only being a demonstration the new thing does not have the same theoretical limits. The question is rather about if such research warrants more eyes and even a negative result can be good because it can communicate that we've found dead ends (which is something we actively discourage, needlessly forcing many researchers to re-explore these dead spaces). There's so much more that I can say and this is woefully incomplete but I can't fit a novel into our comments and I'm afraid the length as it is already results in poor communication to the given context of the platform. Thanks anyone who has taken this time.