News for hackers. Stuff that matters.

A comment on A/B testing gets misused to juice metrics in the short term. Parent comment

Re:A/B testing gets misused to juice metrics in the short term by spookthesunset on Tuesday July 12, 2022 @11:41AM (#32071536)
This is why any good org will make sure to observe all important KPI’s while doing an A/B test. If your “email signup” KPI went to the moon but tanked your “bought shit” metric… you should probably roll back.
[ Parent | on HN ]
Re:A/B testing gets misused to juice metrics in the short term by travisjungroth on Tuesday July 12, 2022 @03:35PM (#32074645)
It's really easy for this to be noise from false negatives. On an A/A test with five guardrail metrics and a threshold of p>0.05, you'll get a false negative 22.6% of the time.
[ Parent | on HN ]