News for hackers. Stuff that matters.
Re:Getting to Statistical Significance on a Shoestring Traff... by jvans on Wednesday September 13, 2023 @12:20AM (#37492140)
Building your own bayesian model with something like pymc3 is also a very reasonable approach to take with small data or data with too much variance to detect effects in a timely manner. This also forces you to think about the underlying distributions that generate your data which is an exercise in itself that can yield interesting insights.
Re:Getting to Statistical Significance on a Shoestring Traff... by AlexeyMK on Wednesday September 13, 2023 @12:37AM (#37492253)
[Author here] Heh - yes but don't, though...

Yes: you could use bayesian priors and a custom model to give yourself more confidence from less data. But...

Don't: for most businesses that are so early they can't get enough users to hit stat-sig, you're likely to be better off leveraging your engineering efforts towards making the product better instead of building custom statistical models. This is nerd-sniping-adjacent, (https://xkcd.com/356/) a common trap engineers can fall into: it's more fun to solve the novel technical problem than the actual business problem.

Though: there are a small set of companies with large scale but small data, for whom the custom stats approaches _do_ make sense. When I was at Opendoor, even though we had billions of dollars of GMV, we only bought a few thousand homes a month, so the Data Science folks used fun statistical approaches like Pair Matching (https://www.rockstepsolutions.com/blog/pair-matching/) and CUPED (now available off the shelf - https://www.geteppo.com/features/cuped) to squeeze a bit more signal from less data.

Re:Getting to Statistical Significance on a Shoestring Traff... by j7ake on Wednesday September 13, 2023 @01:15AM (#37492478)
I love fitting models.

I always say in my profession I will fit models for free, it’s having to clean data and “finish” a project that I demand payment.

Re:Getting to Statistical Significance on a Shoestring Traff... by kimi on Wednesday September 13, 2023 @03:28AM (#37493358)
...and pictures in a format the journal likes....
Re:Getting to Statistical Significance on a Shoestring Traff... by hammock on Thursday September 14, 2023 @10:29AM (#37509520)
That works for a website. Doesn't work as well for direct mail