Hey martian, thanks for the feedback. Your methodology is pretty sound - it looks like we use a similar technique. If you supply a small sample size here, you'll get the message, "There is not enough data to determine a conclusive result".

I also agree. This is effectively 2 lines of R code:

    binom.test(500,2100)
    binom.test(1000,2000)

With a boxplots/confidence interval plots (which are not many more lines). Whether this is useful or not depends on how much people know about statistics I guess.

This calculator makes reasonable assumptions. For example, it doesn't use bootstrapping, and it assumes a normal distribution. I don't see how making these assumptions makes the calculator bad, it just means if you are a stats person who wants to do something different, you'll need to implement your own script. Outside of the HN crowd, there are people who don't code, who nonetheless work with a/b tests, and still need a measure of statistical confidence.

Since a binomial distribution approaches a normal distribution for large N, that's a valid assumption to make.

The arguments in this thread over which is the precise distribution are over analyzing, in my opinion. If I had an experiment that gave one winner with a binomial distribution, and a different winner with the normal approximation, I'd feel little confidence in the results and want to run it again or collect more data. I don't see how that would happen though.

Bootstrapping would be better, sure, but it would be confusing for an a/b test calculator to give you different numbers each time you refresh the page.

You're free to write your own R scripts if you want to do it yourself.

I agree with martian's advice about figuring out a minimum number of visitors you expect to need before analyzing the data, or else you're at risk of biasing the experiment by declaring a winner prematurely.