The only way to tell is to read the papers and judge for yourself, and this is the root of the problem: evaluating candidates based on the number of papers and the impact factor (or some analogous metric) of the venues is much, much easier for committees, than actually reading the papers.
If a panel of respected tenured professors in the field ranked the best works in open access journals each month, or if there was a high profile yearly prize for best open access papers of the year, that could help disaggregate reputational effects from the publication itself.
Bootstrapping this sort of thing is hard. I'm sure Nobels didn't signal as strongly in year one as they do now.
The same for counting quotations, by the way. This may be different in STEM fields, but I'm in the humanities. In the humanities bad papers are sometimes quoted as often as brilliant papers, because everybody wants to correct the authors. I've seen more than one colleague literally make his career by writing a polemic and obviously wrong book (books=often improper peer review) and then sneaking a few apologetic articles in which they half-retracted their claims past the reviewers. What's worse, I know postdocs who are actively looking for such untenable topics that will surely stir up a reaction. We're talking about theses, conjectures or claims that nobody defended previously, because they are so obviously wrong or flawed. I've also met people at the postdoc level who literally replied "Who cares, it's one more publication" when I raised doubts about their main claim in an article and mentioned that it might not 'work' for purely formal reasons (but adding that I'd have to check, to be polite).
I'm sure it's a bit less drastic in CS and the STEM fields, but in the humanities publication counting almost enforces mediocrity and the publication of uninteresting, pointless papers with vastly overlapping topics - and it's completely obvious that it does so.
The situation is particularly frustrating, because the system makes it harder to publish good science. Good science needs larger studies and/or better formal underpinnings, both of which take more time and more effort than duplicating fake papers about fake problems.
Well, enough ranting for today, you get the point.
The onus is on the administrations to get a metric that is gamed by advancing human knowledge, instead of throwing logs at flame wars. That's why they get the big salaries.
In other words, I doubt anyone is explicitly penalized for submitting to PLoS Biology instead of Journal of Neuroscience. However, there is an implicit penalty for not submitting to Nature/Science/Cell.
[As an aside, I really hate the obsession with IF and all these other "quantitative" measures of research quality. They're all incredibly ad-hoc, someone easily game-able, and yet otherwise-smart people take them so seriously.]
Also, the concept of IF being a proxy for rigorous review and work quality is weak at best. Just read a few pages of http://retractionwatch.com/
Particularly for early career people, I suspect most of these "quantitative" measurements are basically reflecting where and wtih whom you worked.
They used to have this nice trick where you hired the son of your friend and your friend hired your son. But at least now there are some non-subjective requirements to be met before anyone can get admitted.
(Disclaimer: I used to work in this field at biomedcentral.com)
Competition for positions in academia has gotten so bad that it doesn't matter. Like Google rejecting perfectly good candidates just because they can, but in Academia even south central Kansastucky state university has enough excellent applicants that arbitrary metrics do more good than harm.
The higher the IF, the more likely it is the researcher trying to do something 'hard' and the more people will scrutinise the results. These two factors alone will lead to more retractions.
Someone publishing something completely uninteresting in a journal no one reads will never be in a position to have to retract their paper, no matter how flawed, because no one will notice and no one will care.
Not to mention the opportunity cost associated with OA fees, which can add up if you're talking about lab startup.
The issue I see is that "Academia" is not the same as "the scientists", there is a larger community, that includes institutions, processes, administrative people, etc.
So, yes, for instance, I would love that we all published our papers in, say, arxive, an we were evaluated by the actual impact of our publications (~ PageRank). But that is not how the funding organizations, hiring committees, etc, are going to do it.
How to do the transition from the current state of things to a better one, and not to commit career suicide? That is the question, and that is what those boycotts are aiming to, I believe.
Some more info here: https://www.authorea.com/123679/d8cErAb_olPC5CvcWWwsig
Isn't it a requirement that all NIH funded work must be published as open access?
However, this is still tagged with the source journal, so it doesn't really get you around the career-related issues.
Public Access guidelines: https://publicaccess.nih.gov/policy.htm
It sorta does that. The "faculty" submit 1-2 paragraph blurbs describing papers. The blurbs often point out the strengths and weaknesses of each paper and give a brief description of how it fits into the broader literature. If you can find a couple of people on there with similar research interests, it can be a great way to get up to speed on something.
Is this solely a product of network effects? Or could there be some truth to the top journals' ability to organise the review process?
Nature/Science/Cell are popular because everyone recognizes their "brand" and associates them with interesting, high-impact science. Thus, it's easy for anyone "Hey, we should fund/hire her because she's got a bunch of Nature papers!" It's not miles from the old "Nobody got fired for buying IBM" bit.
In contrast, it's unlikely that one person--let alone more-on the committee has carefully read your J. Soc. Whatever paper and says "This is well done! Get him instead!"
Nature/Science/Cell have a prestigious brand, so many scientists try to get their best work published there. The journals get first pick of new research to choose from, and as a result getting published there really is a good indicator that your work is impactful and high quality. It's a network effect that can be broken only if enough funding & promotion committees, or top scientists in a field take a stand together.
Something I have wondered though:
What if we separate the "measure of quality and impact" from getting published in the journals? Something like "shadow journals", that doesn't publish and edit any articles itself. Its reviewers would pick the top 30 new papers published in their field each month, and just publish links to them. If this would get established, it could be used by funding and promotion committees, and the brand of the original journals would matter less.
Some journals do this. I've had PhD student friends inform me that when they've published to some (well-regarded) journals, after being accepted, they spent months going back and forth with the editor polishing the article.
That said, these news made my life a lot easier right now. I was considering whether to accept an invitation to review for an Elsevier journal (which I've usually avoided). Now I'm fairly sure I will not.
In the US there is a "Fair Access to Science and Technology Research Act of 2015" (S. 779, H.R.1477) that requires federal agencies to have a "public access policy" if they receive extramural research expenditures of over $100,000,000. https://act.eff.org/action/keep-publicly-funded-research-pub...
Scientific domains with open publishing, like machine learning, have a very fast pace and amazing evolution.
Won't biology as a field be affected severely by all this closeness? I think in the long run the whole field is suffering.
However, I'd also like a job and none of the "top tier" biomedical journals are open access. While I'd like to think that the quality of my work would speak for itself, the imprimatur that one gets from Nature, Science, Cell, and Neuron is worth quite a bit when applying for jobs and grants.
Unless the big names in academia--and the search committees they sit on--publicly and credibly commit to this, you're essentially asking people (many of whom are on short-term contracts making $40,000 a year) to take an enormous career risk.