Re:Harvard ethics prof fabricated multiple behavioral scienc...
by wongarsu on Monday July 10, 2023 @11:18AM
(#36667693)
The ML part of CS is in a sense funding replication: there are a decent number of papers whose only premise is "we compare 5 recent papers on this benchmark (and fill a couple pages with discussion)" or "we made a new benchmark, here's how commonly cited papers compare (plus a couple pages of discussion)"
Outside the high-profile cases it seems accepted norm that papers perform far worse when scored against somebody else's benchmark. The real measure of quality is how big the gap is.
I think that's true for low-tier/low-impact CS papers, but the difference is that literally nobody cares about those papers. The high-tier stuff is easily verifiable (like Tensorflow or whatever) and nobody is writing articles about 5% improvements against a benchmark in some obscure niche scheduling and planning domain.
Outside academic CS people are more scientific about experimental results because it has concrete implications on revenue or spend... but they aren't getting the results from conference papers.