News for hackers. Stuff that matters.
Re:Getting to Statistical Significance on a Shoestring Traff... by charlierguo on Wednesday September 13, 2023 @12:41AM (#37492278)
> Gut Check: Especially if you’re off by quite a bit, this is a chance to take a step back and ask whether the company has reached growth scale or not. It could be that there are plenty of obvious 0-1 tactics left. Not everything has to be an experiment.

This is a key point, imo. I have a sneaking suspicion that a lot of companies are running "growth teams" that don't have the scale where it actually makes sense to do so.

Re:Getting to Statistical Significance on a Shoestring Traff... by bertil on Thursday September 14, 2023 @08:33AM (#37508064)
Everything has to be a test early on, but not every test has to rely on random-split-based statistical significance to make a decision. “Would you pay $20 for this?” is a classic way to judge whether your service has product-market fit, and it’s not about sample sizes, not initially.

Some growth teams are trying more exploratory approaches to find something that resonates with simpler approaches. Others rely on A/B tests. Different profiles, but both are “Growth teams”.

Re:Getting to Statistical Significance on a Shoestring Traff... by quickthrower2 on Thursday September 14, 2023 @08:51AM (#37508281)
How are A/B tests for non-logged-in users doing these days with all the cookie banners, privacy and ad blockers etc. Is this stuff still working even?
Re:Getting to Statistical Significance on a Shoestring Traff... by AlexeyMK on Thursday September 14, 2023 @09:43AM (#37508941)
Yes, but it is _rough_. What actually hurts is "browse on mobile, buy on desktop" type behavior.

Still worth doing, but you end up needing more black magic than you'd like (IP-based assignment, Ad Network-sourced assignment, CDN proxies for Analytics tools, etc).

Working on a separate post about that.