News for hackers. Stuff that matters.
Re:5000x faster CRDTs: An adventure in optimization (2021) by mcqueenjordan on Thursday December 08, 2022 @08:21AM (#33907586)
> The infuriating part was that several people sent me links to the paper and (pointedly) asked me what I think about it. Written up as a Published Science Paper, these speed comparisons seemed like a Fact About The Universe. And not what they really were - implementation details of some java code, written by a probably overstretched grad student. One of a whole bunch of implementations that they needed to code up.

> "Nooo! The peer reviewed science isn't right everybody! Please believe me!". But I didn't have a published paper justifying my claims. I had working code but it felt like none of the smart computer science people cared about that. Who was I? I was nobody.

Although I to some degree understand /why/ this is the way it is, it's quite frustrating the amount of ~unfounded belief people place in "published studies". I'm not advocating to denigrate science, but I think folks should still be skeptical of results (esp. low N or non-reproduced) when they're challenged, and not treat them as Facts About the Universe.

c.f. "The code was peer-reviewed, so the application must be bug-free and 100% correct, right?" /s

Re:5000x faster CRDTs: An adventure in optimization (2021) by cldellow on Thursday December 08, 2022 @12:07PM (#33910307)
In 2012, I was doing NLP stuff with the Stanford NLP POS tagger. They claimed it was highly optimized, and hey, who was I to doubt them? But man, it sure _felt_ slow.

So I did some _very minimal_ profiling and, as a complete outsider, found small, simple changes that increased throughput by 4x without affecting maintainability.

I think this fits your "low N" example -- I'm much more likely to believe something is optimal if you can tell me that multiple (experienced!) people have tried to optimize something and failed. If it's just 1 person, that 1 person better have amazing domain expertise for me to believe it.