News for hackers. Stuff that matters.
Re:Annoying A/B testing mistakes every engineer should know by hnhg on Friday June 16, 2023 @07:45AM (#36354922)
Also, doesn't their suggested approach amount to multiple testing? In other words, a kind of p-hacking: https://en.wikipedia.org/wiki/Multiple_comparisons_problem

Edit - and this: http://www.stat.columbia.edu/~gelman/research/unpublished/p_...

Re:Annoying A/B testing mistakes every engineer should know by throwaway202351 on Friday June 16, 2023 @12:34PM (#36359474)
Yeah, a good AB testing framework would either refuse to let you break things down too much or have a large warning about the results not being significant, but that doesn't always stop the business-types from trying to wiggle in some way for them to show a win.