You can't help but be entertained by the amount of plagiarism that exists. A while back I found out that one of my papers [1] was being sold as a final year thesis via ..a video ad on YouTube [2].

[1] https://ieeexplore.ieee.org/abstract/document/6746236 [2] https://www.youtube.com/watch?v=MFNFScqN47o

Don't students have to defend a thesis any more?

I mean I could buy your paper but I would have to know it by heart and understand it in order to defend it.

At least that's how it went when I was studying applied physics (1974-1977).

If I had just bought a paper I would have had a really hard time in the viva voce.

Edit: by -> buy

Depending on the country, the PhD defense is more or less ceremonial. If you have made it to the defense, you can pretty much not fail anymore.

When I defended my PhD in 2013 at Georgia Tech Chemistry, the first half was the public defense, including Q&A. The second half was closed-door, consisting of just the committee and the PhD student. This second half was known to be more combative and it was common for the committee to request additional work. It was very rare for any one to flat out fail, since the student’s advisor wouldn’t allow them to defend until they had sufficient novel research.

In my program, many of us would strategize for the 30-60 minutes of the closed door grilling. We sought to give our committee members obvious things to criticize with the PhD student having prepared arguments to defend against these criticisms. E.g., I ashamedly included quite a few spelling and grammar errors in the first few pages of the summary section of the thesis (the only part anyone would actually read) and we spent at least 15 minutes on my horrible writing ability.

In general, the main outcome of the closed door portion of the defense was requests for additional work. It was common for committee members to suggest additional things that could “improve” the thesis work. Not surprisingly, many of these suggestions involved applying a committee member's methods, even if not plausibly applicable, so that one would publish another paper citing the committee member’s work. Some students, including myself, would have a job lined up before the defense to timebox the amount of additional work that could be requested.

hahaha I had a PhD defence in 2018 in the UK. Criticisms included:

"I don't trust your maths" "I don't feel this analysis is right, but I can't describe in what way" "you are clearly not very knowledgeable" and many other similar things.

Asking me a question and then before I can open my mouth answering it yourself, and then insulting me for not answering it was the start of the viva defence and it set the tone for the rest.

I was also heavily criticised for not having cited a paper that came out in-between submitting my thesis and the defence, despite this being literally impossible to have done so, and, despite having already had gotten a job in that time, was given limited time to do additional experiments, write whole new chapters, new code, do new experiments, etc. Ended up adding 90 pages of material to the thesis.

In the end I had to quit the job I had just got, because It would have been impossible to not fail my PhD program and keep the job.

Afterword's, in behind closed doors discussions it was revealed that one guy had pushed for almost all of the required extra work deliberately to try to make me fail, because I had done something he could not.

These people should be getting treatment at least on par with what they got with #metoo, why is that not the case?

I mean, personally, I don't really care about reputation of academia, but given that you described a horrible story where somebody basically tried to ruin your career, and given that you decided not to name specific people or institutions, it seems that the whole (UK) academia will have to take the reputation hit for the alleged scandal.

Given that they succeeded in completing the poisoning of my opinion of academia and scientific research there is no career there at all. I say complete because PhDs can feel unpleasant, but that's mostly a result of high stress, difficult work, no security, and complete lack of a social life and interaction with other people for 4 years, rather than science itself but it does set an emotional impression of what research is like that is somewhat difficult to overcome.

I defended my PhD in 2009 at George Mason University (GMU). It had two halves, but they were reversed: the first half was private, the second half was public. I think that's a better system.

In the first half I was privately grilled by my committee. They wouldn't let the dissertation go to a public defense until they'd satisfied themselves that it was fine.

As far as I can tell the public defenses usually look ceremonial at GMU (at least in its engineering department), but they aren't actually ceremonial. Anyone from the public can ask questions at a public defense, so they aren't ceremonial. However, the goal of the (first) private half was to try to make sure that the defender is ready for arbitrary questions (because he understands the material). So it's unusual for the public to ask questions that the defender isn't able to answer. I got some questions I hadn't heard before in my public defense, but I was able to handle them.

You can see my public defense here: https://www.youtube.com/watch?v=QYH18NpsRu8

That's really not the case here in the UK, I know several people who failed their vivas and had to make substantial changes to their thesis. It's not common, because there are intermediate level mini-theses and vivas, and people who do badly in those often don't finish at all, but it does happen.

Yes, in UK mine was terrifying. In all likelihood candidate would get destroyed trying that tactic.

I second that statement. I know at least two people who failed their viva in the UK and ended up getting an MPhil instead of a PhD. In both cases the candidates were told by their advisor not to go with the defense but they decided to go for it anyway. I know several more that got “major corrections„ which meant a lot of extra work and another review six months later. edit: typo

My supervisor had a student who apparently just said "no" as her answer to each critique, 38 times. She got the level of fail which is "never ever come back" - which apparently does exist.

Nobody likes a public humiliation so perhaps the defense event is almost ceremonial but the real defense took place before that in order to get the event itself scheduled?

So that even if people don't fail their defenses often, there might be many that didn't get theirs scheduled at all? This is just speculation on my part.

still the effort to come up with something original is order(s) of magnitude more than copying. it's likely that the potential buyers have enough grounding to understand the papers.

What you propose is actually how it used to function until around the mid-20th century. Journals used to be very permissive when accepting papers with the editor only doing a cursory check to make sure the paper wasn't total garbage.

More info here: https://michaelnielsen.org/blog/three-myths-about-scientific...

That’s how it used to be. The reason behind the shift is the rapid increase in the amount of research output in the past decades. The reason that “arXiv + quality check” won’t work is that the amount of research funding and permanent positions (related) did not increase accordingly. (Think of all the people producing science for low wages as Phds and postdocs and then quitting.) Right now the burden of the (unfair?) selection is mostly on the publishers and their prestigious journals/conferences, which then funding agencies/institutions take into account when funding research/hiring people. If we switch “arXiv + quality check”, that burden will just move to the funding agencies/institutions, but it won’t solve the underlying fundamental problem.

In the US, both academic hiring and proposal reviews are already laborious jobs where committee members are supposed to investigate the candidate/proposal deeply on their merits and potential. It shouldn't be a new burden for them to be unable to take shortcuts.

I think the recent wave of low-impact submissions and co-authorship rings is the result of developing countries trying to simplify that process and tying hiring/promotion/pay directly to publication count and related easy metrics.

Are supposed to, but doing that would require so much more work than is usually being put into it now. Especially in the first rounds, it would be impossible to review all candidates/proposals thoroughly without having staff dedicated to just that. (IMO institutions should have such highly-skilled staff.)

It may be true that the SNR in research output from developing countries is lower, but there is still lots of good science. But essentially no publishers with good reputation. So even with the same SNR, the increased pool of countries producing science would add to the publication pressure.

> Imagine a process where the only criteria are technical soundness and novelty, and as long as minimal standards are met, it's a "go". Call it the "ArXiv + quality check" model.

One possible issue is that researchers usually need to justify their research to somebody who's not in their field. Conferences are one way to do this. So are citation counts. Both are highly imperfect, but outsiders typically want some signal that doesn't require being an expert in a person's chosen field. The "Arxiv + quality check" model doesn't seem to provide this.

> I suspect the bigger problem that CS has is a large percentage of poor-quality work that couldn't be replicated.

As a sort of ML researcher for several years, I agree.

As a fellow ML researcher, I want to add that the lack of code along with the publication makes the problem worse.$BIGGROUP gets a paper whose core contribution is a library, published, and yet they haven't released the code 6 months after the conference, effectively claiming credits for something unverifiable.

I guess this can be different depending on your specific field, but in NLP it really changed for the better in the last few years.

I don't have data, but from subjective experience, 5-6 years ago most papers in major NLP conferences didn't have an associated code repository. Now, the overwhelming majority do.

There are still many other problems, for example a big one is reporting of spurious improvements that can vanish if you get a less lucky random seed. But at least including code is now common practice.

Back when I did a stint at something NLP-ish for my master's, one of the problems seemed to be that, apart from lack of code, the data was also often non-public and specific to the study. That made it impossible to compare different algorithms even as far as the results reported in the publications themselves go because the testing methodology was all over the place and the datasets used for testing various algorithms might have been all different. You couldn't really make much out of the reported results even if you believed the authors reported honestly and had their methodology more or less straight.

I suppose the situation regarding common datasets might vary between subfields and NLP tasks, so maybe I just saw a weird corner of it.

Of course the code was also nowhere to be seen.

Availability of code would of course be even more important, both because of replicability and general verifiability, and also because that would allow you to do a comparison with any number of datasets yourself.

Glad to hear that code availability has been improving.

> There are still many other problems, for example a big one is reporting of spurious improvements that can vanish if you get a less lucky random seed.

Considering that a lot of NLP is at least somewhat based on machine learning, don't people do cross-validation or something?

For NLP, sharing data is a bit of a problem though.

You do a paper showing that problem X can be solved slightly better by downloading and training on a billion tweets.

But you don’t have the copyright to those tweets, so you can’t share data.

> don't people do cross-validation or something

A lot of stable problems comes with a dataset already split into train and test.

> You do a paper showing that problem X can be solved slightly better by downloading and training on a billion tweets.

That's true. Sometimes you might try to tweak the algorithm itself rather than the data, though, or experiment with different kinds of preprocessing or something, and in those cases it would be helpful to be able to do different experiments with shared datasets.

My limited experiences were from around the time deep learning was only about to become a big thing, so it might have been different then. Maybe you nowadays just throw more tweets and GPUs at the problem.

Even novelty is a dubious standard that is often used to discard replications and meta-analyses (not novel), empirical work (no novel theory), reformulations and simplifications of existing research (not novel) and null findings (nothing was found so not novel).

Lolwhat, empirical work is the only thing that can confirm (or rather, refute) the worthiness of a theory !?

Well, then you need something else to use as a basis for tenure decisions.

Also, a "best 10%" conference/journal is valuable -- I have only so much time in the day. There are a few conferences for which it is always a good use of my time to read all the abstracts, plus one or two of the papers that seem most interesting. I can't do that for every conference, or even most conferences, in my area.

So the "best 10%" conference/journal is valuable to the consumers, and the prestiege is valuable to the producers. Therefore I think such a thing would simply re-emerge if you somehow killed it.

I mean there are several journals who publish based only on "technical soundness", Nature's Scientific Reports is probably the most well known one. I don't think it helps, in fact IMO it's detrimental because journals would be even more flooded with publications. Also in the article they talk about the conference getting 10,000 submissions, that quickly becomes unmanageable as a conference to accept.

In deep learning a lot of papers end up only on arXiv, and authors don't bother (or need) to send it anywhere. Or even more, some seminal papers and up only on the author's page (e.g. one introducing GPT-2 https://openai.com/blog/better-language-models/). I am not a big fan of the latter, as it gives problems with long-term availability.

Moreover, the correlation with acceptance and impact in existing, but not that high: https://medium.com/ai2-blog/what-open-data-tells-us-about-re...

Of course, only corporate researchers can rely on not publishing in established journals - as their salary and position does not come from "publish or perish" metrics. Ironically, it means that there is more academic freedom in private companies than in academia.

Bell Labs in its heyday was still reputed to be publish or perish, or patent or perish, without any tenure system either. At my university, we had a previous Bell Labs researcher that said he had been pushed out. As a corporate researcher, there is still high pressure to output something.

At it's best, that's what scientific reports is (in a number of subfields)--a place to put research that is reviewed for technical correctness. There may be debate about how well it succeeds, but I think it's useful. I would prefer to have more papers out where people worked on something, found it wasn't necessarily exciting, but it was solid work and it saves other people time.