And one of the suggested solutions (at ~22:00) seems that it would work if it can be adopted - essentially, have top university administrators evaluate top x papers for hiring/tenure decisions and ignore everything beyond that number. What you measure is what you get; if you measure count, you get a deluge of 'least publishable units', if you measure your top 3 papers, then everyone will focus on quality instead of quantity. A counterargument probably is that it's easier for the administors to measure quantity in a way that seems objective and resistant to any arguments or appeals decisions, and it's far harder to objectively compare quality especially if the candidates are from different subfields of research.
The REF is evaluated by humans. I take 2/3 weeks to read a paper. For good papers I might work on them for 6 weeks + to really get into the technique. How can a REF reviewer consume 5 papers to evaluate an academic? How can they consume 200 papers from 40 academics?
The right thing would be to have the department sumbit it's top 5 papers - 3 reviewers could really see what is going on is a department then.
So - how many IEEE conferences are there!? Also things change, it was much easier to get into Neurips 7 years ago... but it's not necessarily the case that the papers from this year will have as much impact or be as good as the papers from then. And as Neurips itself showed, the peer review process is somewhat random - papers are rejected by different panels meaning that getting in is likely an achievement but also possibly a bit lucky. I don't think that evaluating the papers based on where they were published is a good way to allocate public money.
As a strong concrete example, paper 13 from Ferguson's group at Imperial is probably one of the most important documents for the last 20 years (if you live in the UK or France), it was "published" on a website...
It's not even just about academia in general, it's a universal problem caused by lack of personal accountability through the globalization of talent. The exact same dynamic was shown multiple times in HBO's The Wire. Gaming the metrics. LPU's are no different than 10 minute YouTube videos.
[1] https://youtu.be/DJFKl_5JTnA?t=853