Oh, in that sense I'm clueless too. We all are. What I'm pointing out though is that many people in machine learning research don't really understand what's the point of research, or even what's the point of a research paper. They copy each other's writing style and produce the same kind of low-information-content paper.
There's been a huge influx of new people to machine learning research in recent ish years. I think it started around 2012 and the ImageNet results. I did my Master's in 2014-15 (part-time) and I was surprised by the number of people in the machine learning class I took, and how many of them had nothing to do with computer science or AI, and were instead coming from backgrounds in business or more rarely science. At least I'd expect the people with the science backgrounds to know how to write a paper, but it seems most people who come to machine learning from, say, physics or biology, bring with them a facility with calculus and continuous mathematics, but not much in terms of scientific methodology.
Hinton hints at that in the rest of his comment I didn't quote:
What we should be going for, particularly in the basic science conferences, is radically new ideas. Because we know a radically new idea in the long run is going to be much more influential than a tiny improvement. That's I think the main downside of the fact that we've got this inversion now, where you've got a few senior guys and a gazillion young guys.
Well I guess the "gazillion young guys" need a bit of time to figure out how to write good papers, and review them.
>> I do want to add one thing though. A good paper is hard to recognize. A bad paper is easy, but a good paper may be indistinguishable from a bad paper. This is the "paradox" of research and something people need to take to heart. That is if we want to align our job descriptions with our actual jobs.
That's true, and part of the reason why reviewing is hard work. I still find it very hard to reject a paper. What if I'm the one who doesn't get it?
I don't have a problem citing arxiv works. I mean I have no faith in the official system, so why wouldn't I? But this is a clear indication of an impossible bar to pass and demonstrates the ridiculousness of the system. It isn't just that there's a malicious or dumb reviewer, it is that the other reviewers don't call them out, the AC doesn't call them out, and the metareviewer doesn't call them out. The best case was that broken link person and the AC made them update their response, but this was in the response to my 1 page rebuttal (for 4 reviewers) where they got to say whatever they wanted and I had no chance of response. (Their initial review was literally 2 lines) I don't know how anyone can see this and not think that the system absolutely failed at every level. You're probably unsurprised to hear that this is a common occurrence rather than uncommon.
> the vast majority of the researchers in the machine learning community haven't got a clue what the hell they're doing
I'm well aware. __I__ have no idea what I'm doing, but at least I can tell you the difference between probability and likelihood or that tuning on the test set is information leakage. Research often requires venturing into the unknown and unexplored. That's fine. I don't care if we're all stumbling around in the dark. I do care when people are not just unable to admit it, but unable to recognize this. But that's the classic "tell a lie enough times and you'll start to believe it" situation.
I am in full support of that Hinton quote. This is the first I've heard it (or recall at least), but I often say quite similar things (in fact, just did in another thread). I do mean it when I say that our current system harms us and I'm confident that we won't get to AGI with this system.
> the problem is that peers in peer review lack expertise and knowledge and they can't do their job well.
I won't disagree with this point, but I believe that this is a systematic problem rather than an individual. The system encourages this behavior rather than stamps it out. So in that sense, I think people are doing their job very well. It is just that I don't think their job actually aligns with the intent of the job. Classic case of irony, that the group of people that highly discusses alignment is one of the worst at this. But I guess we shouldn't be surprised given that lately we've seen how unethical we've seen people who write about ethics are.
I do want to add one thing though. A good paper is hard to recognize. A bad paper is easy, but a good paper may be indistinguishable from a bad paper. This is the "paradox" of research and something people need to take to heart. That is if we want to align our job descriptions with our actual jobs.