News for hackers. Stuff that matters.
Re:Successful machine learning models: lessons learned at Bo... by anthony_doan on Monday October 07, 2019 @01:43PM (#21183729)
Experimental design is just a t-test? At least accord to that picture it seems that way. There are no ANOVA or interaction test?

Do websites usually just use t-test only? Like adding one feature at a time?

Re:Successful machine learning models: lessons learned at Bo... by amirmasoudabdol on Monday October 07, 2019 @03:11PM (#21184768)
It's even worst than that. Most of the time the validity of the t-test that they are running is questionable. They are technically running an online t-test and as soon as they find significance they stop. This is fundamentally wrong, and not conclusive at all.
Re:Successful machine learning models: lessons learned at Bo... by smueller1234 on Monday October 07, 2019 @05:25PM (#21186131)
A few years ago when I was still working there, involved in the experimentation tooling among other things, we largely excised that particular behavior. What are you basing your assertion on?
Re:Successful machine learning models: lessons learned at Bo... by anthony_doan on Monday October 07, 2019 @08:03PM (#21187327)
Cheating on a hypothesis test... that's terrible.
Re:Successful machine learning models: lessons learned at Bo... by joker3 on Monday October 07, 2019 @01:46PM (#21183769)
People should be using more than one design, but that's not as commonly taught as it should be. I'm going to give a talk about that to my company's ML group in about a month, and hopefully that'll improve things where I am.
Re:Successful machine learning models: lessons learned at Bo... by jj12345 on Monday October 07, 2019 @01:53PM (#21183876)
Not sure if you'd be willing, but I'd love a quick rundown of the high level takeaways if you'd be willing to drop them here.

Are you talking about more than one experimental design in terms of comparing the exp/control distributions or something else?

Re:Successful machine learning models: lessons learned at Bo... by benmaraschino on Monday October 07, 2019 @04:09PM (#21185401)
Not the OP, but I work on similar problems, albeit in a different setting (healthcare, millions+ of patients). The gist is that you have to bake experimental design into the deployment of your ML model, but in many cases a simple RCT or A/B test just won't cut it. This is largely because when you deploy a model, no matter how sophisticated or accurate, there's no guarantee that it'll actually move the needle in terms of the outcomes you care about—hence you need to run some kind of trial. At the same time, you want to maximize overall utility by not having to allocate more subjects to your control arm (or harmful, or resource-intensive and ineffective treatment arms) than you need to. This latter point is much more of a problem in medicine than it is in other settings, as you can imagine. These considerations point to adaptive designs that balance exploration/exploitation, e.g. those based on multi-armed bandits. Currently working on some cool (in my opinion) variations of MABs that incorporate domain-specific knowledge, so I could talk about this all day!
Re:Successful machine learning models: lessons learned at Bo... by tixocloud on Monday October 07, 2019 @06:13PM (#21186545)
Would be really keen to hear you speak about the subject in greater detail actually. Love having to balance the practicalities of implementing a model in production and validating the outcome while not missing out on utility.
Re:Successful machine learning models: lessons learned at Bo... by joker3 on Monday October 07, 2019 @04:13PM (#21185453)
Do you know a good introduction to adaptive/sequential designs? I'm looking for something along the lines of a textbook aimed at a graduate level seminar.
Re:Successful machine learning models: lessons learned at Bo... by chudi on Tuesday October 08, 2019 @04:17PM (#21196707)
Re:Successful machine learning models: lessons learned at Bo... by Breza on Wednesday October 16, 2019 @01:30PM (#21272968)
Thanks!
Re:Successful machine learning models: lessons learned at Bo... by joker3 on Monday October 07, 2019 @03:04PM (#21184696)
I'm just going to present the absolute basics of the topic. It'll be a high-level overview of something along the lines of chapters 1-4 of https://www.amazon.com/Analysis-Experiments-Chapman-Statisti....
Re:Successful machine learning models: lessons learned at Bo... by jj12345 on Monday October 07, 2019 @03:41PM (#21185114)
Thanks for the reference!
Re:Successful machine learning models: lessons learned at Bo... by huac on Monday October 07, 2019 @06:12PM (#21186536)
say you want to try adding two features, which don't you think interact with each other, e.g. a change to the "pick this room button" and a change to the checkout flow. then, you can randomly assign users to two experiments, independently. your t-test results should then be valid if the two features are independent.
Re:Successful machine learning models: lessons learned at Bo... by anthony_doan on Monday October 07, 2019 @07:36PM (#21187157)
> which don't you think interact with each other,

> your t-test results should then be valid if the two features are independent.

Assuming that your assumption are correct on interaction effect.

You can do a hypothesis test on that assumption while including both factors (the two features). Which will clear away any doubt with a 95% confidence or hire a statistician =).