News for hackers. Stuff that matters.
[dead]
Re:Getting to Statistical Significance on a Shoestring Traff... by AlexeyMK on Thursday September 14, 2023 @05:22AM (#37506680)
Thanks for factorial design! I'll update the post to the proper nomenclature.

The frequentist/bayesian debate is not one I understand well enough to opine - do you have any reading you'd recommend for this topic?

[dead]
Re:Getting to Statistical Significance on a Shoestring Traff... by philipodonnell on Thursday September 14, 2023 @03:06PM (#37513342)
Any similar GitHub for Python you recommend?
Re:Getting to Statistical Significance on a Shoestring Traff... by scottfr on Thursday September 14, 2023 @05:30AM (#37506734)
Disagree on using Bayesian statistics. Frequentist statistics are perfect for A/B testing.

There are so many strong biases people have about different parts about UI/UX. One of the significant benefits of A/B testing is that it lets you move ahead as a team and make decisions even when there are strongly differing opinions on your team. In these cases you can just "A/B test" and let the data decide.

But if you are using Bayesian approaches you'll transition those internal arguments to what the prior should be and it will be harder to get alignment based on the data.

Re:Getting to Statistical Significance on a Shoestring Traff... by eru on Thursday September 14, 2023 @06:03AM (#37506962)
Not necessarily.

You can present your Bayesian approaches in such a way that it's almost independent of the prior. Your output will be 'this experiment should shift your odds-ratio by so-and-so-many logits in this or that direction' instead of an absolute probability.

Re:Getting to Statistical Significance on a Shoestring Traff... by JHonaker on Thursday September 14, 2023 @09:03AM (#37508433)
You have to make almost the exact same choices when you rely on fequentist tools. The main difference is they’re pre-made during the development of the tool, so you don’t get insight into what they are without studying the theory behind the test.
Re:Getting to Statistical Significance on a Shoestring Traff... by miksumiksu on Thursday September 14, 2023 @09:33AM (#37508827)
Fixing dysfunctional decision making by delegating it to "data", what could go wrong? Might as well flip a coin and save the money.