Re:Cool URIs don't change (1998)
by sbierwagen on Friday June 18, 2021 @10:40PM
(#27557271)
Speaking of noticing small effects, Mechwarrior Online was four months into its open beta before anyone noticed that an expensive mech upgrade that was supposed to make your weapons fire 5% faster actually made them fire 5% slower. http://www.howtospotapsychopath.com/2013/02/20/competitively...
Re:Cool URIs don't change (1998)
by parafactual on Friday June 18, 2021 @05:49PM
(#27555385)
This principle seems like it extends far beyond site traffic, especially since something like life satisfaction is much harder to measure.
Yes, it's a kind of slippery slope: as typically set up, changes are biased towards degrading quality. If you run a normal significance test and your null is leaving it unchanged, then you will only ever ratchet downwards in quality: you either leave it unchanged, or you have a possibly erroneous choice which trades off quality for degradation, and your errors will accumulate over many tests in a sorites. I discuss this in the footnote about Schlitz - to stop this, you need a built-in bias towards quality, to neutralize the bias of your procedures, or to explicitly consider the long-term costs of mistakes and to also test quality improvements as well. (Then you will be the same on average, only taking tradeoffs as they genuinely pay for themselves, and potentially increasing quality rather than degrading it.)
Is <10% worth caring about? It's certainly not the difference between life and death for almost everyone; no successful business or website is going to shut down because they had ads, or because they broke old URLs. On the other hand, <10% is hardly trivial, people work hard for things that gain a lot less than that, and really, is defining redirects that hard?