News for hackers. Stuff that matters.
Re:Cracking down on research fraud by martincmartin on Sunday July 26, 2020 @08:37PM (#23961288)
What do you suggest as an alternative?
Re:Cracking down on research fraud by adminprof on Sunday July 26, 2020 @11:16PM (#23961955)
This entire thread is really frustrating for me to read. A lot of these ideas are one-liners from people who have had very little experience with different parts of academia, except maybe being a student. Every single idea here are debated to death by academics already (not just university academics, but funding agencies, academic societies, award panels, journal editors and conference organizers), with changes happening all the time.

If it was this simple that a random person here could come up with how to "solve" academia, we'd have already done it decades ago. The ideas also lack nuance and when you get into the definitions of things (for example, p-hacking), then things become a lot more grey; are you allowed to look at a dataset that you spent 2 years collecting if your first hypothesis does not pan out? The clear cut cases are obvious to everyone, it's the grey area that takes 99% of the time to figure out.

Imagine reading a thread where everyone is proposing "solutions" to software development. It'd go something like "software development is a cesspool and 80% of it fails (see voting systems, MySpace, electronic health records, Theranos. Here's what software companies need to do:" [yes I'm being intentionally stupid to demonstrate how annoying this is]

1) stop releasing before the bugs are fixed. Software and games are rushed out. Companies need to take their time to fix the bugs so the users don't have to encounter them.

2) no more technical debt. Programmers are sloppy and introduce technical debt because they are not incentivized to do high quality programs. [yes, see how triggering that is]

3) cap team sizes. Everyone knows that large teams fail more spectacularly. Gmail and Napster and the original version of Google were made by a group of 4 people. Software teams need to be 4-5 people max.

4) programmers must use a transparency scorecard. Software companies like Oracle and IBM charge ridiculous amounts for their work. They hide costs and cut corners. Programmers should be transparent about the work they are doing each day, what data they access, and which functions they are writing.

These changes need to happen. derp derp

Re:Cracking down on research fraud by nl on Sunday July 26, 2020 @11:51PM (#23962092)
Not sure who downvoted this, but this is 100% on the money in my experience.

People here are writing comments like that funding should be tied to "how systematic, logical, and well-documented the research is" as though these things are correlated to the existing criteria at all.

Much of the criticism here is like someone who knows how to build roads criticising agile software development because it makes it look like no one knows what they are going to build in the end. It's frustrating, and wrong, but the errors are subtle and aggregative.

Re:Cracking down on research fraud by adminprof on Monday July 27, 2020 @02:27AM (#23962676)
Yeah I have no idea how it got to -2, but maybe it was my tone. I was even trying to not call out any specific post or debate any single issue.

The thing about academia is it's full of people who love talking about ways to make it better, and rotate into positions of power where they can change things after a few years.

The solutions are complex, require convincing many different stakeholders (even if they're amenable to the change), nailing lots of detail to make it work right. Because peoples lives and careers are on the line. Reputations of entire fields, the way medical discoveries happen, billions of dollars of taxpayer money, major institutions, etc. are not things you want to hack and discover that whoops, you just incentivized the wrong thing and set back cancer research for a decade.

Re:Cracking down on research fraud by marcus_holmes on Monday July 27, 2020 @02:46AM (#23962746)
So, what's the best proposal or line of thinking coming from all this deep thought about how to solve it?

Because you can't possibly be saying that the current system is the best we can do, or that the problem is intractable.

(I agree with most of your proposals about software engineering btw. We should do more of this).

Re:Cracking down on research fraud by adminprof on Monday July 27, 2020 @12:49PM (#23966858)
It'd be a process: participant in a community in a subfield, listen to ideas that have been tried and outcomes, come up with an idea using those lessons learned and success metrics, build consensus around a new idea, test in a single-instance (like one conference, grant review panel, tenure committee at one university), share lessons to the field, do this for a couple of years to show clear success, expand to multiple events in the field, become an exemplar field for that idea and "infect" other fields

Sounds slow but there's thousands of such experiments happening simultaneously right now. This is how a long of major field-sized changes have happened, like the transition to conferences from journals (which had many initial problems like during tenure review or a lack of quality in reviews), etc. Ideas will lose traction at various stages (for example, there was a movement some time ago to use alpha=0.001 instead of alpha=0.05 for null hypothesis testing, which has been limited to that field or subfield).

Re:Cracking down on research fraud by marcus_holmes on Monday July 27, 2020 @02:05PM (#23967719)
Interesting, thanks.

Is the move to pre-publishing servers (like arxiv) a part of this? How does SciHub (and similar) figure into it?

It does sound slow, and a bit trivial, if I'm honest. Are there any examples you can share of successful experiments that have travelled across field boundaries?

Re:Cracking down on research fraud by mturmon on Monday July 27, 2020 @11:38AM (#23966123)
You're right. Love your example.

HN does not have much actual collective expertise in this area (say: governance structures for technical research), but HN is solution-oriented, so ideas will be proposed. They just tend to be not very good. (See also: HN climate-studies threads.)

I've spent 10 or 15 minutes on this thread, and didn't read any comments actually building on what the OP said. OP's author does have expertise in this - he has been writing a series of good stories for Science magazine in this general area. Sigh.

Re:Cracking down on research fraud by roel_v on Monday July 27, 2020 @04:47AM (#23963307)
Thank you for writing this. Your examples were lethargic. I know I should stay away from the 'academia is broken duh duh' threads here but I get sucked into them anyway. They're always full of armchair experts who don't know anything about universities or research except from being students or at best junior researchers and then think this makes them experts on the matter.

But the article itself had that same vibe for me, especially with how cock sure it is that the 'questionable research practices' are 'fraud'. I mean, come on - someone making up the responses of a 500 people questionnaire; yes that I could call 'fraud'. But not being included as an author on a paper, or being included when you didn't contribute that much? I've been in both situations and in some cases, I was completely fine with it; in others I was a bit miffed (mostly because of the same interpersonal frictions that happen everywhere where people work together) - but in none of those I would call it anywhere near 'fraud' or even 'dishonest'. Yes, there exist people who pay 1 or 2 people to write papers for them and then publish those papers with themselves as the only author. Again, that I would call 'fraud'. But the 99.9% of other cases - not even close. Just like because there is one billionaire underage sex trafficer, doesn't mean all of them are and that 'the system' is 'broken'.

And this is how I, again, got sucked into a completely non-productive 'discussion' that is so far removed from reality so as to be completely irrelevant anyway...

Re:Cracking down on research fraud by webmaven on Monday July 27, 2020 @09:13AM (#23964729)
> But not being included as an author on a paper, or being included when you didn't contribute that much? I've been in both situations and in some cases, I was completely fine with it; in others I was a bit miffed (mostly because of the same interpersonal frictions that happen everywhere where people work together) - but in none of those I would call it anywhere near 'fraud' or even 'dishonest'.

As I understand it, ghosting becomes a more significant issue when it enables the omitted author to peer-review their co-authors' papers (and vice-versa) without disclosure of the conflict of interest.

Re:Cracking down on research fraud by roel_v on Monday July 27, 2020 @09:52AM (#23965076)
shrug Sure, theoretically, but someone who wants to ensure a positive review can just as easily collude with someone who didn't do any of the work. And then still - yes it's possible that a completely bogus paper gets signed off on by a dishonest conspirator, but the editor should see the discrepancy between that and the reviews of the other authors, and dig deeper. But the real problems start in the grey zone; like when you're on the fence between 'reject' and 'major revisions'. There is no 'objective' truth there and quite honestly as an author it's a crap shoot and always has been. It sucks the first few times it happens but none of leads to 'peer review being broken'.
Re:Cracking down on research fraud by nl on Tuesday July 28, 2020 @04:03AM (#23973477)
To add to this - there are plenty of issues with peer review, but someone deliberately avoiding an authorship so they can peer review seems low down the list of real world problems.
Re:Cracking down on research fraud by Chris2048 on Monday July 27, 2020 @12:38PM (#23966734)
> If it was this simple that a random person here could come up with how to "solve" academia, we'd have already done it decades ago.

Just because an idea is simple to come up with, doesn't mean it's also easy to implement.

Also, can you explain to me how your programming analogy works? Because I agree with a lot of it, though it seems I'm not supposed to.

Re:Cracking down on research fraud by adminprof on Monday July 27, 2020 @12:56PM (#23966919)
Sure I guess I didn't make that very clear. Just going down the line:

1) stop releasing before the bugs are fixed. Software and games are rushed out. Companies need to take their time to fix the bugs so the users don't have to encounter them.

- This is a tradeoff between shipping time and bugs. You will never fix all bugs, so it's unrealistic. And it's a business decision in many cases. Even the definition of a bug is tricky, like is a usability issue a bug? So it's just a naive idea.

2) no more technical debt. Programmers are sloppy and introduce technical debt because they are not incentivized to do high quality programs.

- No one wants to create technical debt. Obviously it slows down development later on. Again this could be a business decision. Some programs like one-off data science scripts don't need to fix all their technical debt. Technical debt also accrues naturally (like just changing environment, platform, standards) so it's not possible to aim to not have debt in the very beginning. Hindsight is 20-20 and all that. Saying programmers are not incentivized to do high quality programs is just a blanket naive statement, and depends on the definition of high quality programs.

3) cap team sizes. Everyone knows that large teams fail more spectacularly. Gmail and Napster and the original version of Google were made by a group of 4 people. Software teams need to be 4-5 people max.

- Depends on the type of software. Can't just generalize given a few token examples. Expectations also change over the course of the product.

4) programmers must use a transparency scorecard. Software companies like Oracle and IBM charge ridiculous amounts for their work. They hide costs and cut corners. Programmers should be transparent about the work they are doing each day, what data they access, and which functions they are writing.

- This uses one subsection of the software economy to make a point (as a fallacy). But also some of these measures don't make sense, like some programmers read a lot of code or delete lines, and so the metrics are not generalizable.

In summary, these are ideas that someone who has not done long-term software development would say, or someone who has only had experience with one type of software or company would say. They're not well defined, not generalizable, and don't account for the complex and varied sociotechnical process that software development is.

Re:Cracking down on research fraud by gwerbret on Sunday July 26, 2020 @09:07PM (#23961421)
1) I think the idea of funding "people, not projects" (the Howard Hughes Research Institute's byline; I'm not affiliated with them in any way) is a great one. It would probably involve changing the structure of research programs in other fundamental ways, such as by

2) Limiting the number of graduate students. The current relatively-low barrier of entry into grad school provides cheap, motivated labor for PIs who are trying to stretch their research dollars to the limit. The outcome today is far more PhD graduates than jobs, and for those who get jobs, far too many of them for the research funds that are available. The overall quality of research drops when PIs focus what is fundable, rather than what is important.

3) Shifting the burden of the bulk of research work from trainees to salaried research associates/assistants/lab techs.

4) Change the focus of research output from novelty and volume (of papers published) to quality and significance of work done. Good research is often slow and careful, and doesn't fit well with the demands of grant funding agencies.

Fewer grad students means more resources available for their training, and better career prospects. Shifting the bulk of the research benchwork to salaried professionals removes the incentive to commit fraud. None of this has to come at the expense of research quality; as an example, the NIH/HHMI already have approaches to vetting research programs for quality, even if the PIs aren't competing for grant funding.

Re:Cracking down on research fraud by an_opabinia on Sunday July 26, 2020 @09:11PM (#23961436)
> Limiting the number of graduate students.

Isn't a lot of the graduate school product providing a way for people to immigrate to the United States?

The economy around it is pretty complicated and definitely there's way more pressure to supply more, not fewer, spots.

Re:Cracking down on research fraud by vkou on Sunday July 26, 2020 @09:57PM (#23961657)
No, the problem is a bit more structural then that.

A single professor trains dozens of grad students over the course of their career. Even if you remove every single foreigner, and every single person who does not want to pursue a career in academia, you still end up with dozens of graduates - per professor.

The number of jobs available for those graduates?

One - that professor's - when he or she retires. And until then, they get to burn the midnight oil, doing grunt work on their projects, in the hopes of competing their dozen collegaues for a shot at that one spot.

Re:Cracking down on research fraud by nl on Sunday July 26, 2020 @11:46PM (#23962064)
> The number of jobs available for those graduates? One - that professor's - when he or she retires.

This simply isn't true. Remaining in academia is only one of many career paths for graduate students, and many are financially compensated much better.

Some examples of these include physics and math grad students being recruited by hedge funds, comp sci grad students being recruited by tech companies and geology grad students being recruited by oil, gas and resource companies.

Even within academia the size of the market isn't static. New and emerging universities are hungry for qualified research professors and many come from these programs.

Re:Cracking down on research fraud by vkou on Sunday July 26, 2020 @11:53PM (#23962103)
I already excluded the people who are getting a degree because they want to go to industry with it.

Of the 'stay-in-academia' camp, the ratio is hideously stacked against the graduates.

New and emerging universities aren't doubling the academic jobs pool every five years... Which is what it would take to keep up with the graduates being churned out.

(And incidentally, they are also contributing to the oversupply, because their professors will also be churning out even more new graduates.)

Re:Cracking down on research fraud by voxl on Sunday July 26, 2020 @11:05PM (#23961903)
Your second and third suggestions will literally kill all research at anything but the most prestigious universities. It is elitism. Graduate students are not always RAs, they're usually not. They often teach, so you're not giving an accurate depiction of the incentives placed on them.

Becoming a graduate student is not easy, the bar is not relatively low, and making it through the program is even harder. It's true that academic positions are not large enough but you're ignoring the private sector.

Research scientists could not possibly be paid enough by universities. These would be graduate students minus the mentorship. A worse of all worlds.

Re:Cracking down on research fraud by cscurmudgeon on Sunday July 26, 2020 @08:42PM (#23961308)
Research publication, funding, and jobs should not be tied to outcomes of research.

They should be tied to how systematic, logical, and well-documented the research is. We need a system wide change from funding bodies, job committees to publishing criteria.

Re:Cracking down on research fraud by mlthoughts2018 on Sunday July 26, 2020 @08:46PM (#23961327)
Along the same lines, “what question to ask” or “what idea to try” should never generate credit or fame for individuals. This incentivizes Indiana Jones style research, results at all costs.

As a researcher you should be desiring to get jobs or rewards based solely on the care of methodology, clarity of communication and ease of reproducibility.

If you do those things well and the papers turn up negative results, that’s good archived knowledge for society. Turning up positive results should be viewed as an emergent property of a wide network of labs, agencies, universities and governments, and never a property of darling individuals.

Re:Cracking down on research fraud by dasudasu on Sunday July 26, 2020 @09:15PM (#23961457)
It can be argued that part of being a good researcher is developing an intuition and taste for interesting and promising research questions. Of course, the counterpart to this is that you see hype and fads developing very easily. But overall it is very easy to come up with experiments that any experienced researcher would have told you beforehand had no chance of producing interesting / impactful / positive results. The system must discriminates against researchers that are able to come up with interesting (positive) results with greater probability, since it represents every time an expense of finite resources. If your only contribution as a researcher is to be able to make conscientious and well realized experiments, then sadly your place is much more as an assistant than a PI.

I've never seen it argued in serious research circles that grants should be given this way, but I did find the non-research community to be overly obsessed over negative results not being published or rewarded. It is simply a consequence of the set of possible experiments being infinite, the same way you can come up with an infinite number of startup ideas, but not all of them are equally good even before implementation. Should we reward startup founders with failed startups because they tried their best and really did the best they could given the circumstances?

Re:Cracking down on research fraud by mlthoughts2018 on Sunday July 26, 2020 @11:18PM (#23961961)
The comparison with startups isn’t valid - they seek an explicit profit outcome where good or bad ideas are judged by success in markets. Research is meant to be a public good, that furthers society’s at large repository of knowledge and culture.

> “ It can be argued that part of being a good researcher is developing an intuition and taste for interesting and promising research questions.”

I don’t think this can be argued actually. This is mythology, usually applied to creditmongers who run labs and accumulate accolades that are actually due to a wide array of students and post-docs who are made to get reduced credit as a type of dues paying laced with rampant discrimination and sexism.

I’d say mythologizing the idea of a crack sleuth who has a special knack for research intuition is extremely harmful on all fronts: depriving value to society because it’s false and depriving value to the wide network of lower level staff who are actually responsible for progress.