Re:P hacking – Five ways it could happen to you
by jean_lannes on Monday May 12, 2025 @10:57PM
(#43969312)
These seem like two different things. Testing many different optimizations is not the same experiment; it's many different experiments. The SE equivalent of the practice being described would be repeatedly benchmarking code without making any changes and reporting results only from the favorable runs.
[ Parent
| on HN ]
Re:P hacking – Five ways it could happen to you
by pizlonator on Monday May 12, 2025 @11:41PM
(#43969486)
Doesn’t matter if it’s the same experiment or not.
Say I’m after p<0.05. That means that if I try 40 different purported optimizations that are all actually neutral duds, one of them will seem like a speedup and one of them will seem like a slowdown, on average.
Re:P hacking – Five ways it could happen to you
by daveFNbuck on Tuesday May 13, 2025 @12:28AM
(#43969650)
That's not p hacking. That's just the nature of p values. P hacking is when you do things to make a particular experiment more likely to show as a success.