AlexeyMK

That is mostly right, with the caveat that customer.io is trying to go upmarket too now.

https://playbooks.hypergrowthpartners.com/p/picking-your-lif...

I wrote this up about a year ago for a more comprehensive perspective for companies series A+

If you'd rather go through some of this live, we have a section on Stats for Growth Engineers in the Growth Engineering Course on Reforge (course.alexeymk.com). We talk through stat sig, power analysis, common experimentation footguns and alternate methodologies such as Bayesian, Sequential, and Bandits (which are typically Bayesian). Running next in October.

Other than that, Evan's stuff is great, and the Ron Kohavi book gets a +1, though it is definitely dense.

There's a pretty good book dedicated to this exact question: https://unscalablebook.com/

Yes, but it is _rough_. What actually hurts is "browse on mobile, buy on desktop" type behavior.

Still worth doing, but you end up needing more black magic than you'd like (IP-based assignment, Ad Network-sourced assignment, CDN proxies for Analytics tools, etc).

Working on a separate post about that.

Super helpful - looked it up, will aim to apply next time!

Curious how the bonferroni correction applies in cases where the overlap is partial - IE, experiment A ran from Day 1 to 14, and experiment B ran (on the same group) from days 8 to 21. Do you just apply the correction as if there was full overlap?

Thanks, that is a well reasoned argument!

My take is for small n (say 5 experiments at once) with lots of subjects (>10k participants per branch) and a decent hashing algorithm, the risk of uneven bucketing remains negligible. Is my intuition off?

False positives for experiments is definitely something to keep an eye on. The question to ask is what is our comfort level for trading-off between false positives and velocity. This feels similar to the IRB debate to me, where being too restrictive hurts progress more than it prevents harm.

Thanks for factorial design! I'll update the post to the proper nomenclature.

The frequentist/bayesian debate is not one I understand well enough to opine - do you have any reading you'd recommend for this topic?

> Nooo! First, if one actually works, you’ve massively increased the “noise” for the other experiments

I get that a bunch at some of my clients. It's a common misconception. Let's say experiment B is 10% better than control but we're also running experiment C at the same time. Since C's participants are evenly distributed across B's branches, by default they should have no impact on the other experiment.

If you do a pre/post comparison, you'll notice that for whatever reason, both branches of C are doing 5% better than prior time periods, and this is because half of them are in the winner branch of B.

NOW - imagine that the C variant is only an improvement _if_ you also include the B variant. That's where you need to be careful about monitoring experiment interactions, I called out in the guide. But better so spend a half day writing an "experiment interaction" query than two weeks waiting for the experiments to run in sequence.

> Second, xkcd 882 (https://xkcd.com/882/) I think you're referencing P-hacking, right?

That is a valid concern to be vigilant for. In this case, XKCD is calling out the "find a subgroup that happens to be positive" hack (also here, https://xkcd.com/1478/). However, here we're testing (a) 3 different ideas and (b) only testing each of them once on the entire population. No p-hacking here (far as I can tell, happy to learn otherwise), but good that you're keeping an eye out for it.

[Author here] Heh - yes but don't, though...

Yes: you could use bayesian priors and a custom model to give yourself more confidence from less data. But...

Don't: for most businesses that are so early they can't get enough users to hit stat-sig, you're likely to be better off leveraging your engineering efforts towards making the product better instead of building custom statistical models. This is nerd-sniping-adjacent, (https://xkcd.com/356/) a common trap engineers can fall into: it's more fun to solve the novel technical problem than the actual business problem.

Though: there are a small set of companies with large scale but small data, for whom the custom stats approaches _do_ make sense. When I was at Opendoor, even though we had billions of dollars of GMV, we only bought a few thousand homes a month, so the Data Science folks used fun statistical approaches like Pair Matching (https://www.rockstepsolutions.com/blog/pair-matching/) and CUPED (now available off the shelf - https://www.geteppo.com/features/cuped) to squeeze a bit more signal from less data.

Another option is monetizing via a niche job board for your readers. Pallet.com is good for that - will find companies, etc. Disclosure, tiny investor.

Sure, but you need an "undeleted" view per table now - I haven't seen an ORM offer maintaining those views capability out of the box - any to look at?

You can get around the "check in every transaction" problem with an ORM, but now you're (more) coupled to your ORM which you will occasionally inevitable need to circumvent for something or other. And now you've made it (more of a) leaky abstraction.

It's an article on a personal blog targeting a specific readership familiar with the term.

More broadly, I think the level of granularity at which it makes sense to define terms (Mesos, Kubernetes, even Uber) has to roughly match the level of familiarity of the reader that will get something meaningful from the piece.

This is super well done & polished. Will try out for future blog posts.

Thank you!

> The truth is, had tech workers actually assumed a significant measure of political influence, and led in local politics, San Francisco would today be one of the greatest cities in the world. But not only was such political influence not achieved, it was never attempted.

Is that right? I thought Ron Conway was decently involved in SF politics (https://www.sfexaminer.com/news-columnists/hes-back-ron-conw...) and Ed Lee was backed by tech (https://www.theverge.com/2017/12/13/16771912/mayor-ed-lee-sa...).

My mental model lately has been, if I'm enjoying the content, I should try to figure out how to pay ~$1/hour.

If I'm not enjoying it, I should find a better way to spend m time.