News for hackers. Stuff that matters.
Re:Potential organized fraud in ACM/IEEE computer architectu... by yutopia on Monday June 08, 2020 @05:42PM (#23461131)
Academia has become tremendously cutthroat but it still operates under the assumption that everyone acts honorably.

Perhaps that’s no longer sustainable, but what is the solution? Many of the problems with modern-day academic research can be traced back to increased scale and competition, neither of which is going away.

Re:Potential organized fraud in ACM/IEEE computer architectu... by seesawtron on Tuesday June 09, 2020 @04:01AM (#23464691)
There are a lot of possible solutions that scientific community (Early Career Researchers) is working on.

Open Science: There is a growing movement by ECRs to publish Journals that are open access or atleast offer the option to do so with slightly extra costs. This helps to dismantle the centralized consortia of publishing groups that have become too powerful.

Open review: There are journals that allow the reviewers' and authors' comments to be published alongside the paper. This ensures that reviewers are reasonable in their conduct. See eLife for research in natural sciences[0] where once your paper is under review, it is published no matter what. The reviewers can claim that their issues wer not addressed if they feel so and that gets published along the paper's final version. Similary in the computer science community, there are conferences that follow the open review policies as well.

Competition can be replaced with collaborations if the Group Leaders are open minded and are "raised" in such an environment. The next generation of researchers are today's graduate students.

Publish or Perish: This archaic movement can be suppresed when the universities hire people by looking at CVs of applicants where they can only see the papers but not the Journals where they were published. H-index and traditional metrics can be replaced by Altmetrics [1]. The hiring committees need to do the extra work to go through the research of their applicants and not be biased by the journals in which they publish.

These are only few things that are being done. See OpenScience movement variants[3] to explore more in this direction.

[0] https://reviewer.elifesciences.org/author-guide/journal-poli...

[1] https://www.altmetric.com/about-altmetrics/what-are-altmetri...

[3] https://oa2020.org/

Re:Potential organized fraud in ACM/IEEE computer architectu... by effie on Monday June 08, 2020 @07:39PM (#23462016)
Current U.S./Europe academia seems unable to change from within. It lives on government money which it itself controls and distributes to those who "play ball" - the others are pushed away. The only way a change can happen here if academia-independent people control and distribute money to individuals who showed talent/potential/results in the past. It will take massive outside interference to fix or supplant the corrupt system.
Re:Potential organized fraud in ACM/IEEE computer architectu... by cycomanic on Tuesday June 09, 2020 @03:25AM (#23464531)
That's not quite how it works and in my opinion not the main problem. Take for example DARPA grants, they are controlled by outside people and I don't think it works. You essentially have a "case worker" often with little understanding of the field pushing in some direction. Moreover the amount of admin around these grants is staggering.

The root cause of the situation is that there is not enough money in the system and compared to previously more money is distributed by competitive grant processes. This results in a couple of things.

1. The cutoff (reviewer mark) where grants get funded or not is completely arbitrary, because it is in the flat part of a arctan or logit type curve, so small differences or luck can me you are able to do your research.

2. Competition therefore is fierce.

3. To balance the odds you have to write a lot of applications (academics now spend most of their time either writing grants or administering their grants, not doing research.)

4. It encourages incremental research, because non-incremental research implies high risk, but no one can afford the risk to fail, because having a significant reduction of publications reduces you chances of getting grants in the future (see 1.). Once you get some gap in your funding its almost impossible to get back.

5. This disadvantages women who want to have children.

Re:Potential organized fraud in ACM/IEEE computer architectu... by effie on Wednesday June 10, 2020 @10:29PM (#23484648)
Yeah more university/grant administration of the current sort is not the right way to go. They have too much power already.

Established elder scientists with little bias and little incentive to corrupt the country education/science system should be in charge, not administrative pencil pushers. Maybe hire some outsiders and set up some kind of independent commission of renowned scientists and budget planning/audit people to maintain oversight of the supported science groups.

But lack of money isn't really part of the problem, there is lot of money there and it often goes to waste. Putting in more isn't the solution - it would just amplify the current situation.

The fact there is competition for money isn't a bad thing. In theory it keeps people engaged and working efficiently. The problem with the current competition is that people are judged by their willingness to play ball, by flawed citation metrics, publicity, imaginary department points and relationships. Not by their originality and weight of their scientific contribution.

Re:Potential organized fraud in ACM/IEEE computer architectu... by softwaredoug on Monday June 08, 2020 @08:36PM (#23462382)
It does seem like there's sometimes too much inside baseball in grant review/awards. I wonder if there needs to be a notion of a 'stakeholder' in research that can measure/advocate for the general market of the research outcomes
Re:Potential organized fraud in ACM/IEEE computer architectu... by effie on Monday June 08, 2020 @09:40PM (#23462768)
Yes, the public should be the main stakeholder if public is funding the project. The problem is, it takes an expert to judge the work of another expert. Wise people outside the local academia should represent the public in such things. Maybe one way is to hire and pay well some foreign professors to do the adversarial oversight/evaluation of our local science workforce. For example, a renowned German scientist hired and payed by French government for 5 years to criticize and oversee funding of French scientists close to his area of expertise. The system has to be more adversarial to weed out/defund the bad activities.
Re:Potential organized fraud in ACM/IEEE computer architectu... by wegs on Tuesday June 09, 2020 @05:35AM (#23465115)
1) Reduce competition. Make jobs less desirable to where there might be able to win without cheating. More people at lower salaries would be good here. More research/teaching gets done, and you eliminate subclasses like adjuncts and lifelong postdocs.

2) Ban universities which take federal grants, federal student aid, 501(c)3 status, etc. from using NDA/non-disparage agreements.

3) Put in FOIA/public records-style transparency measures.

4) Open science / open data. Publications should be open too, as should source code. Anyone should be able to replicate scientific research unless it requires $$$ equipment or e.g. builds on personally-identifiable information.

5) Democratic governance processes within universities and salary caps (it doesn't get at this stuff, but it does at a lot of embezzlement).

6) Limit number of grad students per advisor.

Basically, let enough sunshine in that the bad stuff can't stick around.

Re:Potential organized fraud in ACM/IEEE computer architectu... by BeetleB on Monday June 08, 2020 @08:45PM (#23462432)
> Academia has become tremendously cutthroat but it still operates under the assumption that everyone acts honorably.

I think the problem goes a bit beyond that, in that the notion of what constitutes acting honorably has changed.

Re:Potential organized fraud in ACM/IEEE computer architectu... by BurningFrog on Monday June 08, 2020 @07:26PM (#23461904)
Seems like Peer Review is hopelessly broken. Conflicts of interest galore.

Science did fine without it until ~40 years ago, and it can do so again. I hope.

Re:Potential organized fraud in ACM/IEEE computer architectu... by mhh__ on Monday June 08, 2020 @07:51PM (#23462113)
The solution is still within peer review, I feel. What ethical alternative is there?

Required reproduction of the result could be a step forward, but only for cheap research.

Re:Potential organized fraud in ACM/IEEE computer architectu... by BurningFrog on Monday June 08, 2020 @08:37PM (#23462392)
I don't think science was unethical before Peer Review.

As I understand it, modern peer review came about as a 'crowd sourcing" solution to the hugely expanding number of articles to publish, in ever more specialized fields.

The journals couldn't keep enough relevant expertise around, so letting the scientists review each other was the simple fix.

Re:Potential organized fraud in ACM/IEEE computer architectu... by throwawayiionqz on Monday June 08, 2020 @11:30PM (#23463391)
This does not make sense.

Before scientists started reviewing each other, who at the journal was judging the work of scientists to be published? Scientists?

Re:Potential organized fraud in ACM/IEEE computer architectu... by BurningFrog on Monday June 08, 2020 @11:45PM (#23463455)
Well, science literate people working at the journal, I assume.

The point is that they did not have a conflict of interest.

In 2020 peer review, people are often asked to review the work of their rivals. So they're incentivized to (1) say that it's bad and demand many changes, and/or (2) steal their ideas.

Re:Potential organized fraud in ACM/IEEE computer architectu... by wegs on Tuesday June 09, 2020 @05:38AM (#23465132)
arxiv and letting good stuff bubble up after-the-fact.

Peer review in my field is a random process. It's not corrupt; it's just that reviewers give papers a 30 second skim. I've had one paper really reviewed in my whole career. What gets in and out is just random. NIPS did a nice study on this many years back.

On the flip side, academic discourse gets delayed. People aim for Science/Nature, and then down the line. Journals / conferences aim for the prestige of high reject rates. Research goes out sometimes years too late.

Peer review gives the impression of credibility, without the reality. It should be gone. Tiers of journals should be gone too.

Re:Potential organized fraud in ACM/IEEE computer architectu... by tnecniv on Monday June 08, 2020 @09:42PM (#23462778)
Peer review is much older than that. It dates back to the 18th century.
Re:Potential organized fraud in ACM/IEEE computer architectu... by BurningFrog on Tuesday June 09, 2020 @09:59AM (#23466693)
Sure, but it didn't become mandatory and institutionalized until fairly recently.