News for hackers. Stuff that matters.
Re:Harvard ethics prof fabricated multiple behavioral scienc... by justrealist on Monday July 10, 2023 @10:44AM (#36667208)
> My experience in CS is that the replicability of experimental results is embarrassingly bad

I think that's true for low-tier/low-impact CS papers, but the difference is that literally nobody cares about those papers. The high-tier stuff is easily verifiable (like Tensorflow or whatever) and nobody is writing articles about 5% improvements against a benchmark in some obscure niche scheduling and planning domain.

Outside academic CS people are more scientific about experimental results because it has concrete implications on revenue or spend... but they aren't getting the results from conference papers.

Re:Harvard ethics prof fabricated multiple behavioral scienc... by wongarsu on Monday July 10, 2023 @11:18AM (#36667693)
The ML part of CS is in a sense funding replication: there are a decent number of papers whose only premise is "we compare 5 recent papers on this benchmark (and fill a couple pages with discussion)" or "we made a new benchmark, here's how commonly cited papers compare (plus a couple pages of discussion)"

Outside the high-profile cases it seems accepted norm that papers perform far worse when scored against somebody else's benchmark. The real measure of quality is how big the gap is.

Re:Harvard ethics prof fabricated multiple behavioral scienc... by UncleMeat on Monday July 10, 2023 @10:51AM (#36667322)
I'm talking about papers in top PL and Security venues.