News for hackers. Stuff that matters.
Re:Collusion rings threaten the integrity of computer scienc... by version_five on Wednesday May 26, 2021 @05:51PM (#27296551)
The bigger issue here, and what is threatening the integrity of the research, is blind reliance on conference publication count as a proxy for research quality.

Maybe it's time to move on from some of these conferences, and focus on interactions that maximize sharing research findings. I know that is unrealistic, but like every other metric, conference acceptance ceases to have value once the metric itself is what people care about.

Re:Collusion rings threaten the integrity of computer scienc... by psychomugs on Wednesday May 26, 2021 @05:59PM (#27296640)
I often go back to this talk from Michael Stonebraker about, in his terms, the diarrhea of papers [1]. It's difficult to justify time towards anything that doesn't lead to a near-term publication or another line on your CV.

[1] https://youtu.be/DJFKl_5JTnA?t=853

Re:Collusion rings threaten the integrity of computer scienc... by PeterisP on Wednesday May 26, 2021 @06:57PM (#27297194)
A really awesome talk - it's nowhere close to my field of research, but it is oh so relevant anyways, thank you for linking it and I'd recommend everyone interested in this larger topic to listen to what Stonebraker had to say on this.

And one of the suggested solutions (at ~22:00) seems that it would work if it can be adopted - essentially, have top university administrators evaluate top x papers for hiring/tenure decisions and ignore everything beyond that number. What you measure is what you get; if you measure count, you get a deluge of 'least publishable units', if you measure your top 3 papers, then everyone will focus on quality instead of quantity. A counterargument probably is that it's easier for the administors to measure quantity in a way that seems objective and resistant to any arguments or appeals decisions, and it's far harder to objectively compare quality especially if the candidates are from different subfields of research.

Re:Collusion rings threaten the integrity of computer scienc... by anonymousDan on Thursday May 27, 2021 @04:27AM (#27300602)
This is what the UK REF system does effectively. You get evaluated on at most your top 5 papers.
Re:Collusion rings threaten the integrity of computer scienc... by sgt101 on Thursday May 27, 2021 @05:08AM (#27300798)
The top 5 in the last 4 years - which is pretty lunatic when you think about it. A Ph.d - which is meant to extend human knowledge by 1 iota - takes 3 years to write. So what does 5 papers indicate?

The REF is evaluated by humans. I take 2/3 weeks to read a paper. For good papers I might work on them for 6 weeks + to really get into the technique. How can a REF reviewer consume 5 papers to evaluate an academic? How can they consume 200 papers from 40 academics?

The right thing would be to have the department sumbit it's top 5 papers - 3 reviewers could really see what is going on is a department then.

Re:Collusion rings threaten the integrity of computer scienc... by anonymousDan on Thursday May 27, 2021 @10:22AM (#27303489)
REF is 7 years I believe. 2/3 weeks per paper - I'm not sure what stage of your career you are at but I can't see how this would be necessary to evaluate the quality of a paper ( which presumably has already passed peer review and been published in a top tier venue ). Also, a PhD is in effect a trainee researcher. I don't think it's unreasonable to expect a more experienced academic to have a higher output rate.
Re:Collusion rings threaten the integrity of computer scienc... by sgt101 on Thursday May 27, 2021 @11:11AM (#27304050)
>which presumably has already passed peer review and been published in a top tier venue

So - how many IEEE conferences are there!? Also things change, it was much easier to get into Neurips 7 years ago... but it's not necessarily the case that the papers from this year will have as much impact or be as good as the papers from then. And as Neurips itself showed, the peer review process is somewhat random - papers are rejected by different panels meaning that getting in is likely an achievement but also possibly a bit lucky. I don't think that evaluating the papers based on where they were published is a good way to allocate public money.

As a strong concrete example, paper 13 from Ferguson's group at Imperial is probably one of the most important documents for the last 20 years (if you live in the UK or France), it was "published" on a website...

Re:Collusion rings threaten the integrity of computer scienc... by anonymousDan on Thursday May 27, 2021 @11:47AM (#27304471)
No comment on the quality of many IEEE conferences :) I think most people active in a field know what the best venues are. I certainly don't disagree that there is a lot of randomness to the review process. And frankly picking which papers are likely to be most influential in ten years time is super difficult (e.g. see how often best papers have minimal subsequent impact). Regarding the Ferguson document, I haven't served on a REF review panel, but I think if you can justify it as being influential/high quality through other means that should be acceptable?
Re:Collusion rings threaten the integrity of computer scienc... by sgt101 on Thursday May 27, 2021 @12:21PM (#27304839)
OFC you are right - it would be fairly career limiting to try to talk it down I would think! But I am concerned that the REF should work by the reviewers actually looking at the material, rather than looking at where the material was published...
Re:Collusion rings threaten the integrity of computer scienc... by indeedmug on Wednesday May 26, 2021 @07:58PM (#27297667)
This is a great lecture. While he is specifically talking about database research, a lot of what he is relevant to academia.
Re:Collusion rings threaten the integrity of computer scienc... by ItsMonkk on Wednesday May 26, 2021 @09:19PM (#27298247)
"When a measure becomes a target, it ceases to be a good measure." - Goodhart's Law

It's not even just about academia in general, it's a universal problem caused by lack of personal accountability through the globalization of talent. The exact same dynamic was shown multiple times in HBO's The Wire. Gaming the metrics. LPU's are no different than 10 minute YouTube videos.

Re:Collusion rings threaten the integrity of computer scienc... by mycologos on Wednesday May 26, 2021 @08:40PM (#27298001)
I made a similar comment in another thread here, but the problem is that there will always be non-experts who want to assess the quality of research being done in different fields. These might be deans, department heads, VPs of R+D labs, and so on. Ideally, all research would have clear measurable impact, but a lot of good research doesn't, at least not immediately, so there needs to be some way for non-experts to measure quality. Conference/journal publications and citation counts are highly imperfect solutions, but I'm not sure what the better candidates are.
Re:Collusion rings threaten the integrity of computer scienc... by jonathanstrange on Thursday May 27, 2021 @05:37AM (#27300951)
Assessment can never be done by non-experts. They do not understand the subject matter and cannot even judge which journals are good. It just makes no sense.

When someone is hired (whether for tenure or time-limited), their research as a whole has to be evaluated by external, independent committees who take into account the content of the research and do not base their judgment on indicators only. There is no shortcut around that.

The biggest annoyance nowadays is the decision-makers' insistence on "excellence", though. You cannot have only excellent people everywhere, as per the definition of "excellent", yet this demand is in every fucking guideline for postdocs and tenure-track position. It's absolutely ridiculous.

Re:Collusion rings threaten the integrity of computer scienc... by goatinaboat on Thursday May 27, 2021 @05:58AM (#27301056)
Assessment can never be done by non-experts. They do not understand the subject matter and cannot even judge which journals are good. It just makes no sense.

The problem with this assertion is: non-experts are paying for all of this.

Imagine you are an ordinary taxpayer. You are feeling the pinch yourself, you look around you and see infrastructure crumbling, every day the press says healthcare and this and that is underfunded. Now along comes some scientist, he or she wants a few billion for a new particle collider that will make no difference whatsoever to your life, and their only justification for it is "well other scientists say we should get all this money, and they're cleverer than you, shut up".

Can you see why funding something with no accountability might be considered problematic?

There's also the fact that an expert who can't explain something to a non-expert probably doesn't understand it very well themselves...

Re:Collusion rings threaten the integrity of computer scienc... by jonathanstrange on Thursday May 27, 2021 @07:45AM (#27301845)
If non-experts are in charge of individual funding, then you're virtually guaranteed to get nonsensical decisions. It couldn't possibly work, and even if you got it to work, the results would be abysmal.

> Can you see why funding something with no accountability might be considered problematic?

Maybe you're confusing budget decisions with funding and hiring decisions. These are fundamentally different. Universities and research institutions, as well as national funding authorities, get budgets that are decided politically, i.e., by elected representatives. These can have broad categories and guidelines or preferred research areas (e.g. "excellence initiatives"). Budgets are usually allocated well in advance, for instance our national funding authority gets budget security for 4 year periods (if I'm not mistaken). How they spend it is dictated by political guidelines for the respective period and plenty of complicated national and international laws.

In contrast, I was talking about hiring decisions and decisions about individual funding. How can it not be obvious to you that these decisions need to be made by experts on the basis of CVs and scientific project proposals, not by politicians or other laymen?

Re:Collusion rings threaten the integrity of computer scienc... by chalst on Thursday May 27, 2021 @06:35AM (#27301266)
Only experts can make this judgement in the short term. As time goes by, the reception of the work accumulates and it becomes increasingly possible for non-experts to judge.

It took over 20 years after the Standard Model reached broad acceptance (ie., the experts thought the theory was probably right) for the first supercollider powerful enough to observe the Higgs boson to be financed. This was enough time for policy makers to reach high confidence that the experts had not gone badly wrong and for there to be reasonably well-informed public opinion on the merits of the search.

Re:Collusion rings threaten the integrity of computer scienc... by eek04_ on Thursday May 27, 2021 @03:29PM (#27307118)
The better candidate is spending more on the evaluation.

E.g, for my company's twice-yearly evaluation, everybody writes up a short report on the most impactful stuff they've done including evidence, this is evaluated by their manager to give a score, and then there's a series of group meetings between managers to make sure that the scores are calibrated, including looking at all types of metrics that can be dug up and comparing to our written role descriptions for different levels. It takes a lot of time but creates fair scores.

This is extremely labor intensive, but that's the thing: To create anything resembling fair evaluation of a large group of people that do a large set of different things, you need to do things that are labor intensive. Using a simple set of metrics don't cut it.

Re:Collusion rings threaten the integrity of computer scienc... by nradov on Wednesday May 26, 2021 @09:14PM (#27298211)
The department head at least should be something of an expert, and be able to consult with more specialized experts. Then that department head can pass on an evaluation to management.
Re:Collusion rings threaten the integrity of computer scienc... by sgt101 on Thursday May 27, 2021 @05:10AM (#27300813)
Most experts (who understand what a department head is) don't want to be department heads. In the UK there used to be a tradition of making someone be head of department for 2 years or so and then letting them become a professor again so that the next junior could do the head of stint! This has faded because of much nastier politics (everyone hates you after you sacked their friends) and a rise in admin due to the industrialization of the bigger departments (its not a job for an amateur any more)
Re:Collusion rings threaten the integrity of computer scienc... by BlueTemplar on Thursday May 27, 2021 @08:53AM (#27302462)
Looks like it could work if you had to do it in a different university ? (Also, less cronyism.)
Re:Collusion rings threaten the integrity of computer scienc... by geraltofrivia on Wednesday May 26, 2021 @06:18PM (#27296798)
The playing field where one can share research findings is far more uneven, more tilted in favor of the prestigious labs and individuals more than the current field of conferences and the blind review process.

Findings and papers from well known individuals (read: twitter accounts) do get far more attention , more citations. Of course, one can argue that, broadly, well known labs and individuals are wel known because of their tendency to do great work, write better papers. And that’s true. However, the above still holds, in my experience as PhD student in ML. Anecdotally, I have seen instances where a less interesting paper from a renowned lab got more attention and eventually more citations than a better paper accepted at the same venue by a less renowned lab on the same topic.

Re:Collusion rings threaten the integrity of computer scienc... by cycomanic on Thursday May 27, 2021 @02:49AM (#27300155)
I completely agree with you and this is a significant issue, because it makes it hard for someone not from one of the established places to get research attention.

I would also argue that with the increased importance of the (mostly) commercial "high-impact" journals this has become worse. I know that some of the professional (non-expert) editors of these journals specifically look at the citation counts of authors before accepting to send them out to review, because their main aim is to get people reading the articles, not necessarily good science.

Re:Collusion rings threaten the integrity of computer scienc... by geraltofrivia on Thursday May 27, 2021 @03:52AM (#27300448)
This is objectively worse. Commercial journals need to be bygone. I feel that the publishing venues in the field of machine learning are a great example of non commercial, (relatively) transparent, open access publications. Look at ICLR, ICML, ACL, NeurIPS, for instance. All of their proceedings are open access. Most papers their are available on arXiv as well (which has its pros and cons both that I'd rather not get into here). Many of them also have an open review process wherein the reviewers post their reviews for all to see in a forum like interface, allowing frequent back and forth with the authors. In the best case, its more akin to assist the authors refine their potentially sound findings, a dynamic process where the paper may get revised a couple times before acceptance.

I believe journals need to adapt the openreview process. There are obvious parallels in the lifecycle of a journal submission (from submission to acceptance) and the openreview process, except that the latter is accelerated (for better or for worse).

Re:Collusion rings threaten the integrity of computer scienc... by cycomanic on Thursday May 27, 2021 @06:44AM (#27301330)
Yes, the cloud that the large commercial journals have over the academic process is really worrying. A publication in Nature can mean the difference between tenure and no tenure or getting a grant or no grant.

I have to say that I find the review process of the copernicus journals very interesting. You can see a description here: [1]. Unfortunately I don't work in a related field otherwise I would have published there already.

[1] https://www.atmospheric-chemistry-and-physics.net/peer_revie...

Re:Collusion rings threaten the integrity of computer scienc... by BlueTemplar on Thursday May 27, 2021 @08:58AM (#27302513)
Yeah, I was wondering : has someone tried to compare citations of papers by researchers that do not tweet / blog to those who do ?
Re:Collusion rings threaten the integrity of computer scienc... by geraltofrivia on Thursday May 27, 2021 @09:34AM (#27302888)
Yep.

> We find considerable evidence that, overall, article citations are positively correlated with tweets about the article, and we find little evidence to suggest that author gender affects the transmission of research in this new media

https://journals.plos.org/plosone/article?id=10.1371/journal...

(I've only skimmed the paper, a few months ago).