The non-free journals don't do shit except charge rent for their reputation, and print paper copies -- the free journals don't have the prestige as the non-free journals, so count less for your career. I am thinking in my field that Demography and Policy and Development Review lead to tenure, but Demographic Research doesn't lead to shit. This is changing, but not fast enough.
Academics have no means of evaluating their peers other than articles in prestigious journals. They don't make anything that anyone wants to buy, and really no one outside of your subfield of like 100 people will ever read you anyway (100 if you are lucky...) Popular popularity is actually a mark against you, so you will never be measured by sheer number of readers.
(It's a warped system.)
Also, as a researcher, when somebody is asking you for your paper, just send it. In all the cases I really needed a paper and was unable to get it for free, I contacted the author and got it. Even better, it makes you in contact with the author and can lead to collaboration down the line. Old school networking with groups and circles, it works very well...
You mean Google Circles! Sorry, I've been brainwashed...
I would much rather see original research managed OSS style -- published directly to communities of interest, critiqued and discussed openly, with revisions and explanations and discussions and raw data and source code and everything in all its ugly glory right there with it.
There is a temptation under the current system to regard the contents of journals as absolute truth on account of their curation behind closed doors. We used to feel that way about newspapers, too. The truth is that credibility comes in degrees, and it's much safer to be able to see the guts of the operation.
While programmers like coding, researchers hate writing (they like doing research!). While there are relatively few OSS projects, there are millions of research papers (yes, millions! http://scholar.google.com/scholar?q=research&hl=en&b...)
There are wiki style projects out there for each major domain. For example, quantum computing has http://www.quantiki.org. These sites are great, but they will never replace journal articles because they do not handle history or citations well. Imagine if all we had was wikipedia to tell us about the Church-Turing hypothesis? Papers like that give us an insight into the scientists' mind that nothing else can.
The reason it works is that your peers read your work, and either refute or cite you. Remember -- everybody knows everybody in a subfield, or at least everybody's advisor. It helps when there is math involved which is either correct or not. If somebody random submits something to arxiv, people know they don't know him or his grad school and that is thus probably a crackpot and so don't bother to read it (except sometimes someone from nowhere gets a result, which would NEVER be seen in the closed peer review process).
EDIT: Also you attach your name to a finished product, not an unattributed ongoing wiki page. Your reputation is everything, so you win or lose based on the articles.
One of the problems in social science academia is that 98% of the articles are completely worthless, and they only serve to get a professor promotion based on an index of citation times journal prestige. The smaller state colleges require their junior faculty to "publish", but they don't have enough to say to get into real journals and the publishing houses fill the need with "Journal of Rural Sociology" etc.
Of course, publications from the "big guns" go straight to the big journals, and their peer review is more like rubber-stamp if you have the name on your author list.
So the only difference is for the bureaucrats, who want you to show them "proper" papers to be able to move forward, the people doing the actual research only care whether your experiment and explanation is solid or not, can be published anywhere.
(And I'm totally for open journals, even more, use CC-BY on it...)
There are not actually relatively few OSS projects -- like with papers, there are just relatively few that are interesting or important. For example, Github alone claimed to be hosting 2 million repositories. I don't know what fraction of those are private, but . . . if you go looking for OSS projects, there are a lot of them out there.
Most of them -- like most papers -- are not of much interest to anyone (possibly not even the author), and very few are of interest to everyone. I am able to navigate the landscape, though, by observing which projects attract the interest of others, whether through contributions or communities of interest such as this one.
I guess what I am saying is, I like the community acting as a gatekeeper. I think it's more reliable, and it has more brainpower to spend on the job, than a small number of dedicated editors. It's my read that this is the way research works anyways -- when I was in grad school for mathematics, a paper wasn't a proof until most of us had read it, discussed it, and critiqued it among ourselves. My community, not the journal, was the gatekeeper. I think things would be healthier if that were more open and obvious.
To make a change to someone else's article, all you can do is question their methods, and question them again more sternly. It's not trivial to repeat experiments with scientific rigor, but it is (generally) trivial to test out a code change or two.
Another issue is experience - it takes a lot of education to get someone to the point of being able to apply scientific rigor; there's a lot of seductive pitfalls along the way. It's not really something you can be 'amateur' about. That's not really the case with a lot of OSS, where you can successfully contribute without understanding important but subtle issues.
If anything, it's the development behind closed doors that you can "be amateur about". Work done in public is too embarrassing and too unlikely to attract any interest to do a bad job on.
But that's kind of beside the point. Even if the expertise is rarer on research, I don't understand why that matters. Wouldn't that be an argument for pooling the resources you do have? I mean, if there are only four people working in a field, wouldn't it be better if they all could critique a new work, and if they were able to watch it progress and influence its direction (maybe stopping mistakes early!), and if the judgement of a new work came with their written opinions on its worthiness attached?
but in short: It seems you misunderstand me. They both require understanding, but it is easier to effect changes on an OSS project than it is to make an author go back and rewrite or retest their article.
Then you're simply increasing granularity...instead of judging papers by what journals considered them worthy, judge by what individuals considered them worthy...and if you don't know the individuals involved, the reputation system is there to help you.
It'd have to be something similar to pagerank, not a simple vote-count method. High-reputation individuals should be able to contribute more to the reputation of other individuals.
Another idea would be to apply pagerank to article citations, though this would only be useful for older papers.
Pagerank patent expires in 2018. There are some other interesting reputation system out there, too.
I think that's actually pretty similar to what we're doing now. Articles accepted by journals of high reputation get a high reputation, and the journals select high-reputation individuals to do reviews. A good distributed reputation algorithm could perform essentially the same function. Leaving review open to anyone could help preserve net objectivity even if some individual reviewers are biased.
But anonymity could be accomplished by releasing articles initially with a timestamp and a digital signature by a new public key. After review, the authors could reveal that they have the matching private key.
On the other hand, modern science is incredibly political. When millions of dollars of funding is on the line, tensions run high.
Anonymity is one way to shield reviewers. But the only way to have workable anonymous review is a trusted gatekeeper—and those are expensive.
Science aims for truth
While there's a lot of overlap, there's a fundamental difference between the two, and methods to pursue pragmatism are not necessarily the right methods to pursue truth
The point is not so much about the quality of what does come out, it's about the quality of the crap that currently does not come out. While I agree that it could be useful to have open discussions about papers, I think if that happens at the expense of the ability of everyone who thinks they have an idea to put stuff out there, the result might be a net decrease in signal/noise.
That said, this should be pretty cheap since they only need the scientific editor to do this, so a "journal" that only publishes on the web may have a viable (but perhaps not lucrative) business model funding itself purely on page charges.