News for hackers. Stuff that matters.
Re:The Low Base Rate Problem (2014) by tqi on Thursday April 14, 2022 @11:46AM (#31028489)
While this is all true, I think the reason why people so often run underpowered tests is not because they "lacks a firm understanding of statistical power" but because we have not made a compelling case for an alternative. UXR surveys? In my experience that is even more underpowered than AB tests. User testing? Often just as subjective as gut calls.

Gut calls (ie product sense) seem like the only real choice, but I think there really needs to be an argument made for WHY that is the way, rather than just against other options.

Re:The Low Base Rate Problem (2014) by xpe on Thursday April 14, 2022 @12:15PM (#31028975)
Yes, in some cases, teams want an experiment to back them up.

If a decision maker cannot dedicate the necessary resources to properly power a test, it arguably would be more rational to use another process to make the decision. Perhaps a thoughtfully designed team discussion or even a 'Person P will make the decision because the uncertainty is relatively high, consensus is relatively low, and additional discussion is unlikely to clarify -- meanwhile we could go ahead and try the thing and see what happens!"

Re:The Low Base Rate Problem (2014) by tqi on Thursday April 14, 2022 @01:11PM (#31029925)
> If a decision maker cannot dedicate the necessary resources to properly power a test

Apologies if this is what you meant, but in a lot of cases it's not a matter of not dedicating enough resources so much as there aren't enough resources (ie visitors, users, posts, etc) to detect reasonable effect size. And in those cases, it seems like the author's point is that running a test is actively worse than a gut call or focus group or whatever else. But to your point, product teams often don't like gut calls because they feel subjective and unilateral, so having a test allows some degree of shared accountability for decisions.

Re:The Low Base Rate Problem (2014) by xpe on Thursday April 14, 2022 @03:55PM (#31032449)
Well said.
Re:The Low Base Rate Problem (2014) by xpe on Thursday April 14, 2022 @12:09PM (#31028858)
I tend to notice when people write 'the only choice' or 'the only real choice' when they probably mean something more like (1) an option that, when examined, seems best; or (2) an unexamined (sometimes dogmatic) choice. In some cases, a person may have a third intent: by linguistically limiting a choice set, they can frame or limit a discussion (for good or ill).
Re:The Low Base Rate Problem (2014) by tqi on Thursday April 14, 2022 @01:06PM (#31029834)
I dunno about all that, but in my case what I mean is "the only choice that would satisfy the author's criteria of being statistically sound"