That sounds problematic
> there isn't that sort of public stricture as a random field in chemistry. Feel lucky.
It's pure noise over here. I don't feel lucky, I feel frustrated. I get 3 reviewers and 3 completely different reasons to reject, often nonsensical (not joking, recently was rejected because a broken reference link to the appendix). Then submit to another conference and it is another 3 reviewers with 3 different reasons to reject that are completely different from the previous ones. Often asking me why I didn't cite arxiv papers that were released after the submission deadline. ACs don't care as long as reviewers agree because they gotta keep that acceptance rate low. It's playing the fucking lottery except your graduation and career depends on it despite people frequently saying conferences don't matter they'll still critique on lack of top tier publications.
Your system doesn't sound great and also sounds frustrating, but I'll trade you. Regardless, I don't have faith in either system to significantly determine if a work is good (I do believe they can identify bad works, just not good works).
Could one problem with these kinds of reviews be that the reviewers have absolutely zero skin in the game? Like, you can write complete nonsense and nobody will blame you personally for it, you have no stake in the outcome of the submission/paper what so ever, and you're much more incentivized to come up with reasons to reject than to say "good enough". I realize the latter part can be good in theory, as it sets a high bar, but it also often feels like this goes awry when not kept in check somehow.
Actually, it is worse. They probably have skin in the game, and incentives to reject! Most CS fields publish through a conference system. This is a zero or one shot system (if you get a rebuttal phase, _if_ you want to count that). These conferences have acceptance rates that they "need to maintain" to keep their rankings. In other words, you are competing against all other papers being submitted to the same conference, not just papers with similar topics. Even if it is only a little, rejecting a paper actually increases the odds that your submission makes it through.
But yeah, you're in the right ball park. This is also why you will quite frequently see conference review guidelines blatantly violated and why you will see area chairs and metareviewers just not care. They all have incentives to encourage rejection, let alone be impartial.
I think the system works when the community is small and there is accountability among peers. Accountability creates a dampening effect on bad behavior. But at scale, you only need a few bad actors to setup a feedback system and to not just spoil the entire barrel, but the entire shipment.
The people who review your papers in conferences and ask you why you didn't cite future arxiv papers are the same people who put their work on arxiv and cite each other's preprints. You can't rely on the process of "peer-review" on arxiv any more than you can rely on the conference peer-reviews because they're performed by the same people, and they're people who don't know what they're doing.
The sad truth is that the vast majority of the researchers in the machine learning community haven't got a clue what the hell they're doing, nor do they understand what anyone else is doing. The typical machine learning paper is poorly motivated, vaguely written, and makes no claims, nor presents any results, other than "our system beats some other systems". As to reproducibility, hell if we know whether any of that work is really reproducible. Everybody who references it ends up doing something completely different anyway and they just cite prior work as an excuse to avoid doing their job and properly motivating their work. The people who write those papers eventually get to be reviewers (by sheer luck), or sub-reviewers. They have no idea how to write a good paper, so they have no idea how to write a good review, either. And they couldn't recognise a good paper if it jumped up and bit them in the cojones.
I love to cite Geoff Hinton on this one:
GH: One big challenge the community faces is that if you want to get a paper
published in machine learning now it's got to have a table in it, with all
these different data sets across the top, and all these different methods
along the side, and your method has to look like the best one. If it doesn’t
look like that, it’s hard to get published. I don't think that's encouraging
people to think about radically new ideas.
Now if you send in a paper that has a radically new idea, there's no chance
in hell it will get accepted, because it's going to get some junior reviewer
who doesn't understand it. Or it’s going to get a senior reviewer who's
trying to review too many papers and doesn't understand it first time round
and assumes it must be nonsense. Anything that makes the brain hurt is not
going to get accepted. And I think that's really bad.
So the problem is not arxiv or not arxiv, the problem is that peers in peer review lack expertise and knowledge and they can't do their job well.
So I'd ask you do one of two things, either:
- Review a work with the diligence and care that you wish someone would give to you
or
- Don't review
I'd also appreciate it if you openly recognized how stochastic the system is and that if/when you become in a position where you need to evaluate someone, that you remember this and take it into consideration. It has a lot of value to you too, since if the metric is extremely noisy it doesn't provide you value to heavily rely upon that metric. Look for others.
But, people pay me to do a job. It's not in my contract in any clear terms, but to do my job well I need to do all those things; and I like doing my job well.
I don't have a problem citing arxiv works. I mean I have no faith in the official system, so why wouldn't I? But this is a clear indication of an impossible bar to pass and demonstrates the ridiculousness of the system. It isn't just that there's a malicious or dumb reviewer, it is that the other reviewers don't call them out, the AC doesn't call them out, and the metareviewer doesn't call them out. The best case was that broken link person and the AC made them update their response, but this was in the response to my 1 page rebuttal (for 4 reviewers) where they got to say whatever they wanted and I had no chance of response. (Their initial review was literally 2 lines) I don't know how anyone can see this and not think that the system absolutely failed at every level. You're probably unsurprised to hear that this is a common occurrence rather than uncommon.
> the vast majority of the researchers in the machine learning community haven't got a clue what the hell they're doing
I'm well aware. __I__ have no idea what I'm doing, but at least I can tell you the difference between probability and likelihood or that tuning on the test set is information leakage. Research often requires venturing into the unknown and unexplored. That's fine. I don't care if we're all stumbling around in the dark. I do care when people are not just unable to admit it, but unable to recognize this. But that's the classic "tell a lie enough times and you'll start to believe it" situation.
I am in full support of that Hinton quote. This is the first I've heard it (or recall at least), but I often say quite similar things (in fact, just did in another thread). I do mean it when I say that our current system harms us and I'm confident that we won't get to AGI with this system.
> the problem is that peers in peer review lack expertise and knowledge and they can't do their job well.
I won't disagree with this point, but I believe that this is a systematic problem rather than an individual. The system encourages this behavior rather than stamps it out. So in that sense, I think people are doing their job very well. It is just that I don't think their job actually aligns with the intent of the job. Classic case of irony, that the group of people that highly discusses alignment is one of the worst at this. But I guess we shouldn't be surprised given that lately we've seen how unethical we've seen people who write about ethics are.
I do want to add one thing though. A good paper is hard to recognize. A bad paper is easy, but a good paper may be indistinguishable from a bad paper. This is the "paradox" of research and something people need to take to heart. That is if we want to align our job descriptions with our actual jobs.
Oh, in that sense I'm clueless too. We all are. What I'm pointing out though is that many people in machine learning research don't really understand what's the point of research, or even what's the point of a research paper. They copy each other's writing style and produce the same kind of low-information-content paper.
There's been a huge influx of new people to machine learning research in recent ish years. I think it started around 2012 and the ImageNet results. I did my Master's in 2014-15 (part-time) and I was surprised by the number of people in the machine learning class I took, and how many of them had nothing to do with computer science or AI, and were instead coming from backgrounds in business or more rarely science. At least I'd expect the people with the science backgrounds to know how to write a paper, but it seems most people who come to machine learning from, say, physics or biology, bring with them a facility with calculus and continuous mathematics, but not much in terms of scientific methodology.
Hinton hints at that in the rest of his comment I didn't quote:
What we should be going for, particularly in the basic science conferences, is radically new ideas. Because we know a radically new idea in the long run is going to be much more influential than a tiny improvement. That's I think the main downside of the fact that we've got this inversion now, where you've got a few senior guys and a gazillion young guys.
Well I guess the "gazillion young guys" need a bit of time to figure out how to write good papers, and review them.
>> I do want to add one thing though. A good paper is hard to recognize. A bad paper is easy, but a good paper may be indistinguishable from a bad paper. This is the "paradox" of research and something people need to take to heart. That is if we want to align our job descriptions with our actual jobs.
That's true, and part of the reason why reviewing is hard work. I still find it very hard to reject a paper. What if I'm the one who doesn't get it?
> That sounds problematic
Bingo, it is problematic. It's part of how "the Big boys" get their academic and compensation (investment in their firms) rewards. You have to play the publish-or-perish, paper rank game to get ahead and stay ahead in academia, and this leads to all sorts of problems. Authors don't want to question "the Big boys" because that leads to their papers not getting published because the "the Big boys" and their bootlicker wannabes are the reviewers and they will exact their tribute. Make it to "the Big boys" club and now you're a gatekeeper and now you're also responsible for perpetuating this system.
It's why Nature-style peer review needs to become a thing of the past.
I'm not saying that "popular" (for a value of "popular" that involves peers at large, not the public at large) peer review is / will be without problems. But it seems to me that it will -at least for a while- be less corruptible.
The other hill I will die on is that we shouldn't refer to journal/conference publishing as "peer review." This is one form of peer review, but there are MANY more. And as far as I'm concerned, 3 randos that briefly skim my paper in an adversarial setting (zero sum) looking to reject works barely constitutes peer review. Peer review is what happens when your peers read your work, test it, build upon it, replicate it, etc. We need to stop this language because it helps no one.
I would add: when your peers' review commentary is public.
If journals posted first-submission comments for published papers I think there's be a radical change in comments. For good or ill, I have no idea.
Much as giving credit is important, this is unfortunately also how you end up with intro sections littered with references to the same ten papers everyone's read already just to back up some extremely vague/general statement.
No, this isn't a survey paper. So it is serving no purpose other than greasing their peers.
I'm a chemist, who has considerable established competition. So our reviwers know the systems and studied the method in grad school and sweated over it as a postdoc and had to innovate for cash in a professional capacity while doing something other than what we were doing.
They'd excoriate for vague methods, or poor explanations, or general nonsense. You had to explain everything and make sure to refence the Big boys you were close to but definitely not stealing from.
I think ML is unformed enough that there isn't that sort of public stricture as a random field in chemistry. Feel lucky. We don't even get to hide behind LaTeX, because there's too much benchwork for anybody's boss to ever give a fuck about it.