News for hackers. Stuff that matters.

A comment on Why small tweaks and split testing don't work, and what to do instead. Parent comment

Re:Why small tweaks and split testing don't work, and what t... by jonathankoren on Friday November 30, 2018 @02:23AM (#18567375)
You need to make sure you have enough samples in order to know if you rejected the null hypothesis by chance. Stopping your test early, is a form of p-hacking. See:

https://heapanalytics.com/blog/data-stories/dont-stop-your-a...

[ Parent | on HN ]
Re:Why small tweaks and split testing don't work, and what t... by closed on Friday November 30, 2018 @07:30PM (#18574112)
Peeking at your data, and calculating the sample size you need for a test are separate statistical issues. I agree that peeking messes up significance levels :).

The point I was trying to make was you can decide to run a test with a very small sample (e.g. n = 5), and it will still have the level of type 1 power you set if you chose a significance level of .05.

> You need to make sure you have enough samples in order to know if you rejected the null hypothesis by chance.

You do this when you decide the significance level (e.g. .05). The value needed to reject, given a significance level, is a function of sample size.

The definition of Type 1 error on wikipedia has a good explanation of this:

https://en.wikipedia.org/wiki/Type_I_and_type_II_errors

[ Parent | on HN ]