In one of the neurology labs I was in at UCLA, I saw the same exact thing: publishing tons of garbage and not doing real science. There were some professors of higher integrity, but the PIs that were doing the garbage publishing were getting millions of dollars.

One of my PIs, who was doing this, had a term for his publishing strategy: "salami publishing". You try to cut as thinly as possible your research and publish as many papers on it as you can. That's actually less harmful than the real problem which is that the papers were just rubbish and people in the lab seemed to all kind of know, but disliked discussing it.

Yup, the circle-jerk is strong.

My PI at Janelia once dismissed a proposal I had for an 'unsexy' but pragmatic tool that probably everyone in the building would have used. I was told it would 'do nothing for his career'.

That was pretty depressing.

The official term is MPU - minimal publishable unit.

Do we need an anonymous but authenticated web of trust system for analyzing/debunking papers? Is it apparent to those in the field that these results are low quality / junk science?

How does one detect junk science paper other than being skilled enough to have produced the bad science if the first place?

It’s definitely apparent to those in the field. All the good scientists know who publishes junk, even if the junk publishers are unaware. Arjun Raj has a great and relevant blog post on playing the long game in science: http://rajlaboratory.blogspot.com/2016/06/reproducibility-re.... Worth a read.

Is "salami publishing" even harmful at all? Isn't a larger number of smaller papers analagous to a larger number of smaller modules in software?

I don't think the analogy is valid. To be honest I hate it when I encounter a researcher who seems to be taking this strategy.

Large parts of these papers are boilerplate or duplicated between papers. This wastes the time of the reader even if they try to identify where the new parts are. Plus, the papers are published in 5 different journals or conferences and might not even cite each other, so you often aren't even aware of all of the pieces.

In my opinion, the whole idea of "papers" isn't going to stay. Papers was a way to share information when internet didn't exist. In a future, researching agents (most probably AI or robots) will "chat" and "argue" to each others, building fact and fact-checking each other sentences.

It would basically be the classical reviewing process in real time on every sentences.

In my opinion, "salami" papers go in that direction. Facts, premises and results are decoupled to more units, which can be independently claimed and proved.

I agree we need to move to a more continuous approach. We should constantly update a (digital) model of our understanding of a topic, and each (attempted) update should reevaluate against all existing cases.

Current papers are too fragmented, and primarily accumulate knowledge in those humans that actively participate in the field. By them reading the papers, evaluating and combining them. To transfer that knowledge to someone new is a process measuring in multiple years (ie a PhD). And when a person then ultimately exits their field, a lot of the accumulated understanding is lost.

Great points. I agree that a more granular approach would be better. But I think "salami" papers are too poorly implemented at present.

As a compromise in the meantime, it might be better to encourage more short papers which avoid the boilerplate parts. This would make salami slicing your papers more acceptable as you wouldn't be expected to write a full length paper, which encourages salami slicers to pad their papers.

The French "Comptes Rendus" journal works like this, and I consistently find the papers I've encountered in it to be an efficient use of my time.

The whisper network I may (or may not) be privy to, may (or may not) indicate that LA's Chem is terrible when it comes to #MeToo.

If you're not willing to give an allegation the weight of your own semi-anonymous word, and it's only an allegation, not even an anecdote...

What the hell are you even writing?

I did my graduate school in engineering and saw many of the same problems. Projects were often done simply because the matched the "brand" of the lab, meaning that they used some technology that had been developed in the lab. The experiments were almost always done haphazardly with only good results being published. Additionally, any barriers that were encountered were overcome by de-scoping the paper (removing claims that require breaking through the barrier). At the end of the day, at least basic science contributes some amount of information (however small) to the body of knowledge. Engineering research can just produce waste because it doesn't even bother to search for a fundamental truth.

Edit: I think there is still a role for engineering academics. The most successful ones that I have seen are those that are motivated by chasing down specific scientific questions and developing technology that will enable them to find answers.

In an ideal world, any well formed experiment is a source of new information. You are supposed to have valided or rejected an hypothesis, and by sharing it, other people can build on that knowledge. We don't live in an ideal world, and people well-being, including rewards, feelings and hopes, must be placed before the ideal goal.

Of course, the usual solution to that issue is to formalize and robotize the process of research, which is probably what the world is slowing tending to.

You might be interested in the Ronin Institute, basically a bunch of former academics who left traditional institutions, but are still pursuing scientific research. http://ronininstitute.org/

I'm still a grad student, but am definitely considering pursing my own goals once I finish. Fortunately the finical barriers are not insurmountable for some really interesting fields.

Thanks for the link. Seems valuable for independent researchers who want to collaborate with others like themselves.

>Join me. Quit your job, start a company.

When you're an international student on the F1 visa, it's not that easy. The moment you're out of school, you have to get back to your home country. Basically, you can swallow it and shut up for 5-n long years, or go home dishonored and waste at least n years (n = number of years you have been in grad school) of your life.

Either way, you're fucked.

I sympathize. My cofounders are international students. The pressure is on be in a place to do H1Bs for them before they graduate.

> I'm going to do science. But on my own terms.

Hey, I'm interested in this. I already quit my (non-science field) job and started a company. :-) But I'm just getting started on science. I'm using scientific principles, but I want to advance my practices.

From what I can tell so far, measurement-based or evidence-based science (hard science) is inextricably linked with the concept of society, because "that good which can be packaged and successfully transmitted across a network of social bonds" is really the main goal measuring systems serve.

Unfortunately, measurement science is also extremely shallow by its objective nature and facilitates what might be called "information pornography" when it has become too far removed from the subjective realm. Because measurement is also the art of using labels for things, the definition of (and blind spot of) "science" within this kind of institution I'm reading about would lean hard toward "labeling".

It seems that no matter how far we try to outrun the subjective, qualitative realm, our caricatures of it are really what haunt us most about it. So to sum up, what I'm looking for is reasonable measurement-based science _in society_ linked up with deep, high-quality subjective science.

Does any of this ring any intuitive bells with you? I am looking not for yin to win over yang, but for ways as a _non-professional scientist_ to meaningfully combine these aspects--to link up with others who are willing to philosophize, theorize, take measurements, remain open to new ideas, etc. within a framework that is not so swayed by some of the shallower measuring incentives, but still meets reasonably high standards.

Would love to get your thoughts--thanks for your comment and sorry to hear about your experiences at the institution in question.

In case anyone thinks this is an isolated idea, I also left academia and the scientific laboratories to work independently. While the exact reasons for doing so are multifaceted, I'll echo that there's a lot of questionable, finely parsed research that I disagree with.

Alright, so how does it work? For me, I'm on contract consulting for a few months a year and the rest of the time I spend doing research. Now, there are number of difficulties with this approach. Namely, it helps to work in a field that generates enough money to support myself for the rest of the year. Though it varies by field, I believe this has been far easier for me since I carry a Ph.D, which adds credibility on contracts. Second, I don't get to simply do research the rest of the year since it's also important to do product and client development to stay in business. Third, while consulting can pay well, it's also volatile. I've been in business full time for more than five years, so it has worked well enough, but I probably would not operate in the same manner if I had a family. That said, I've been able to look into and master many things that I would have never been able to while working in a traditional academic environment.

In short, I've been able to make this work by consulting to buy back my time. I do think there are other models to making this work and if anyone else has been successful doing independent research I'd love to hear how.

Great comment. I'm a final year PhD student right now who is planning on doing something similar. If you wouldn't mind, could you email me at the address in my profile here? I'm interested in learning more about your approach.

In addition to consulting I'm also considering the following:

- being an expert witness (a type of consulting)

- translating papers and books (worked for Julian Barbour during the cold war, but I'm not qualified and don't suspect there is great demand)

- being a lecturer or adjunct at a university (In my field it's about 10K USD per class best I can tell. My goal with this would be to help smooth out your income if consulting is too volatile. Also gives you access to university resources.)

- writing technical books (unlikely to be enough on its own, but I'm looking more into people who have done this)

- teaching workshops/short courses (Works for Edward Tufte. If books can be pirated easily then it might be best to give away the PDFs for free, sell hard copies, and make most of the money with workshops.)

- science journalism (probably a dead end)

- open source (D. Richard Hipp seems to have an interesting approach)

- donations/crowdfunding (probably not sufficient but worth looking more closely at)

- small grants (my advisor recommended this)

A better approach is to draw attention, start new institutions, do independent research, and the like. As with most revolutions, burning things down wastes energy and just creates a vacuous wasteland that is often filled with all manner of vermin. Successful, new institutions would demonstrate that a viable alternative is possible. The idea is that by demonstrating something better, the defective ones will need to change their ways or suffer extinction.

> Join me. Quit your job, start a company. Burn it all down.

And hopefully, be the owner of the next broken institution (aka: Google, Facebook, ...).

I totally agree with you, but I see this as a simple cycle.

Maybe. But this is really an engineering problem.

Albeit one we currently lack the tools to seriously tackle.

It's ridiculous that politicians and not physicists and complex systems theorists set policies in our society. We need to start viewing the economy like we view biology and understand on a deeper level the effects policies we put in place will have _before_ enacting them. Corporations are literally organisms and have agency.

It's not rocket science - if bribery is legal and corps can improve their bottom line by changing the law they _will_. They must. Or a competitor will and put them out of business.

Your politician _can't_ serve your interest, he has to keep being a politician first.

The world is pretty scary right now - corporate interest is unchecked and in danger of fucking up our societies for good.

This is how the world crumbles. Not with a bang or a whimper but by everyone 'just trying to get by'.

Pitchforks people. Pitchforks.

I don't think replacing politicians with scientists and engineers is really the solution here. We will always need the interpersonal skills that politicians bring to the table. However, these politicians should really employ more scientists and engineers as advisers. Most of the nitty gritty policy is currently handled by aides. These aides should have more training in evidence-based policy.

I'm not suggesting replacing politicians with scientists. I am trying to motivate that policy making must be approached quantitatively, and that the calculus for doing so must be developed. Definitionally this will be done by scientists.

Design the incentives right for politicians and watch as self-organization works its magic.

They are already defined, we have a lot of economics and politics theory. The problem is, the scientists, the engineers and the technical people in general don't like and don't study this kind of knowledge, because they think the politicians should be switched for technicians.

That's not true. The comparison to biology is very literal here. We simply do not have the tools to really understand complex systems.

This is where our physics fails us. We're great at predicting the behavior of systems with a small number of interacting degrees of freedom. But for systems with a huge number of interacting degrees of freedom (everything around you) we're hooped - the math is intractable to existing techniques. New approaches are required.

There _are_ some overarching principles for complex systems (namely self-organization). But they have not been well-developed.

I think it's Moloch [1] - people behaving based on incentives of a system that nobody really controls and exists largely because of poor coordination.

It isn't easy to fix because nobody has total power over the system in order to fix the bad incentives.

[1] http://slatestarcodex.com/2014/07/30/meditations-on-moloch/

I'm with you. Want to chat? https://invisible.college

The invisible college was started in the 1600s by Robert Boyle (of Boyle's law), as an alternative to the corrupt academy of his time, which cared more about authority than reason and evidence. The invisible college went on to invent peer review.

Now peer review has spread throughout the academy! But corruption has exploited its weaknesses and we need to build a new system, outside the academic walls, to usher in the next era of science.

Please talk with us at the new invisible college. We have a new system for peer review. It isn't yet published but it's ready for discussion.

[email protected]

I was thinking about starting a forum for independent researchers, but looks like you beat me to it.

Can't say I'm a fan of websites that require JS to do anything, unfortunately, but I'll check it out more closely when I get the time.

We're currently working on the forum software. I'd love to hear your ideas. Send me an email? [email protected]

If the client-side JS is a dealbreaker, let's talk about it -- we can build a server-side renderer.

Sure, I'll send you an email this weekend. I have a list of ideas from a few months back, actually.

Wonderful! I'm really looking forwad to it.