Confidence.js – make sense of your A/B test results

Re:Confidence.js – make sense of your A/B test results by jessicaraygun on Monday March 31, 2014 @04:48PM (#7504052)

Hey! I'm the author and I'd be happy to answer any questions you might have.

Confidence.js is based on the A/B testing code that we use at sendwithus.

A/B testing math is hard and we've worked really hard on making this great. Let us know what you think!

[ Parent | on HN ]

Re:Confidence.js – make sense of your A/B test results by sharnett on Monday March 31, 2014 @05:15PM (#7504237)

A nit pick: I believe your 'readable' interpretation of the confidence interval isn't quite right: http://en.wikipedia.org/wiki/Confidence_interval#Meaning_and...

[ Parent | on HN ]

Re:Confidence.js – make sense of your A/B test results by jessicaraygun on Monday March 31, 2014 @05:44PM (#7504409)

Hi sharnett,

Thanks for bringing that to my attention - I agree that our explanation is a little more convoluted than it needs to be. I'll update that later today to be more "readable" ;)

[ Parent | on HN ]

Re:Confidence.js – make sense of your A/B test results by teleclimber on Monday March 31, 2014 @05:41PM (#7504386)

Thank-you for sharing this.

Can you recommend a resource that explains the Math for people who slept through statistics class?

I actually need to brush up on this topic for a new project so anything you can recommend would be greatly appreciated.

Thanks!

[ Parent | on HN ]

Re:Confidence.js – make sense of your A/B test results by jessicaraygun on Monday March 31, 2014 @05:54PM (#7504471)

Stats class R sleepy... Real world applications like A/B testing are definitely more interesting.

We did a lot of digging and there aren't a lot of resources that are easy to digest - but this one really stood out to us:

http://visualwebsiteoptimizer.com/split-testing-blog/what-yo...

I'd love to write a summary of everything I've learned - maybe a blog post in the near future!

[ Parent | on HN ]

Re:Confidence.js – make sense of your A/B test results by tagawa on Monday March 31, 2014 @11:16PM (#7505960)

This slide deck worked for me, with the explanations in understandable chunks: http://elem.com/~btilly/effective-ab-testing/

[ Parent | on HN ]

Re:Confidence.js – make sense of your A/B test results by mattmanser on Monday March 31, 2014 @05:25PM (#7504297)

I think making it an object has made the code unnecessarily complicated. I think you could almost 1/2 code with a simple function.

This is where I personally think javascript programming is going wrong at the moment, so much boiler plate.

You're also programming a bit too defensively. Ask yourself, do you honestly need all those checks? What would actually happen if you got rid of the checks? I think your code would be clearer without them and the programmer would still realize he'd made a mistake and passed in nulls.

[ Parent | on HN ]

Re:Confidence.js – make sense of your A/B test results by jessicaraygun on Monday March 31, 2014 @05:43PM (#7504398)

Thanks for the feedback!

Confidence.js comes from how we operate on A/B tests at sendwithus - we think of them as variants associated with results. While the code could definitely boil down to a few simple math functions, we find objects to be a good way to organize our data models.

[ Parent | on HN ]

Re:Confidence.js – make sense of your A/B test results by hiisi on Monday March 31, 2014 @07:24PM (#7505061)

Also, "getRequiredSampleSize" is called twice for each variant - in the "getResult"'s loop and then inside "hasEnoughData". Same thing for "getVariant". Would you welcome a pull request?

Your project is great, I'd totally use at work. Also, it was to learn the math behind that. Thanks!

[ Parent | on HN ]

Re:Confidence.js – make sense of your A/B test results by jessicaraygun on Monday March 31, 2014 @07:29PM (#7505102)

Glad you like it, happy to have you contribute :)

[ Parent | on HN ]