No, we don’t. Increasing demands for rigour in pre-publication peer review are why publication times from submission sociology and economics reach and exceed two years and why papers which start off thirty pages long end up with eighty, after adding robustness checks and citing every tangentially relevant paper in the literature. We know that post publication peer review works perfectly well because it was the norm until after WWII with the rise of state funding of big science and the accompanying ass covering and form filling proceduralism that made it popular.
As always only replication counts, whether that’s checking that an experiment has the results claimed or that an argument follows from its premises.
We certainly don’t need to lean on reputation more. Science isn’t law; arguments from authority aren’t valid.
I felt it personally in grad school. If you objectively observe the work of yourself and your peers in such an environment, you may notice that there's a reason that none of you got into the Ivy Leagues.
The publication+review+reproduction process is fine to discover scientific fact, I totally agree.
You know if there's anything written about the history of the modern scientific process, specifically on the rise of state funding of science? I'm particularly interested in when academics started to be essentially required to bring in external funding. I've only read offhand remarks like this and don't feel I have the full story.
How?
> As always only replication counts
It's too bad replication studies have such a hard time getting funded.
The idea is that you post your paper on some common repository like arxiv. The "reviewer entity" or RE which is self-organized group with interest in sub-field can be invited to review it. The RE accepts or rejects it. Citations graph propogates to RE as reputation score.
The problem with his proposal:
1. Authors are not anonymous which can bias prestigious REs to avoid papers from unknown authors.
2. Everything still depends on citations, most of which are usually worthless as people mostly cite to fill obligatory related sections, not because they are actually using your work.
3. No obligation for REs to review anything. Doing reviews is tiresome and busy people may just avoid it unless there is a clear obligation/honor system involved.
4. Prestigious REs will be invited to review by everyone, causing highly uneven distribution for the workload.
I believe openreview system was started based on LeCun's thoughts but needless to say we haven't found a good system that can resolve above issues. More importantly, any change in system needs to come from community leaders who are part of organizing commitees for conferences or website developers like arxiv and Google Scholars. Unfortunately, last two have been virtual stasis for long time.
It's astonishing how little investment exists for our main engines of progress that is scientific progress. Compare number of developers working on arxiv or Google Scholar vs Twitter!
I think the model is still good. The examples you stated really do fail at the reproduction part. I think one of the difficulties here is that reproduction is difficult and costly, and there is no one willing to pay for it. A common metric for a paper is "can a smart PhD student replicate the experiment from the contents of the paper?" How many advisors ask their students to replicate a paper? How many students can?
Interesting, to say the least, though I can't say I have time to read all 150 pages at this very moment.
On another internship I worked for a researcher who bragged about having published almost 10 papers on one of his algorithmic discoveries without ever revealing totally (which allowed him to start a for-profit company with it)
You know, the whole publication + review + reproduction thing really helped science become a more solid process, but we need something more elaborate now. Probably some kind of reputation system that would not just be the number of citations by friends and colleagues.