News for hackers. Stuff that matters.
Re:Superhuman Built an Engine to Find Product Market Fit (2018) by dwallin on Thursday August 29, 2024 @10:26AM (#41391164)
Is it just me, or is this bad statistics?

With a small sample size and large numbers of personas / categories you would expect to see a positive bump, even if there was no statistical relationship between the persona and the preference. Since you are only eliminating categories that don't happen to be represented in the subset you are testing, you can only ever actually go up.

For demonstration I rolled 20 dice randomly for 6 personas and 3 categories of preference:

1, 5, 4, 4, 4, 2, 6, 1, 4, 2, 1, 2, 5, 6, 3, 6, 3, 2, 1, 3

A, A, A, A, B, C, B, A, C, A, C, A, B, C, A, A, B, A, B, B

A = 10, B = 6, C = 4; Which gives me 20% for C

I restrict myself to just the numbers that voted for C (2, 4, 1, 6) removing all 3 and 5s

I now am left with:

1, 4, 4, 4, 2, 6, 1, 4, 2, 1, 2, 6, 6, 2, 1

A, A, A, B, C, B, A, C, A, C, A, C, A, A, B

This now gives me A = 8, B = 3, C = 4

And now I get 27%, a nice 35% boost! Even better than Superhuman's 10% boost. But this is all an illusion, there was absolutely no dependency between the persona and preference here, which you would only see with a large enough sample size.

Re:Superhuman Built an Engine to Find Product Market Fit (2018) by Nevermark on Monday September 02, 2024 @06:50PM (#41429438)
The pudding proof is the percentage of “very disappointed not to have” in the next round of customers, after adapting customer targeting and product from the previous customer batch.