News for hackers. Stuff that matters.
Re:Successful machine learning models: lessons learned at Bo... by joker3 on Monday October 07, 2019 @01:46PM (#21183769)
People should be using more than one design, but that's not as commonly taught as it should be. I'm going to give a talk about that to my company's ML group in about a month, and hopefully that'll improve things where I am.
Re:Successful machine learning models: lessons learned at Bo... by jj12345 on Monday October 07, 2019 @01:53PM (#21183876)
Not sure if you'd be willing, but I'd love a quick rundown of the high level takeaways if you'd be willing to drop them here.

Are you talking about more than one experimental design in terms of comparing the exp/control distributions or something else?

Re:Successful machine learning models: lessons learned at Bo... by benmaraschino on Monday October 07, 2019 @04:09PM (#21185401)
Not the OP, but I work on similar problems, albeit in a different setting (healthcare, millions+ of patients). The gist is that you have to bake experimental design into the deployment of your ML model, but in many cases a simple RCT or A/B test just won't cut it. This is largely because when you deploy a model, no matter how sophisticated or accurate, there's no guarantee that it'll actually move the needle in terms of the outcomes you care about—hence you need to run some kind of trial. At the same time, you want to maximize overall utility by not having to allocate more subjects to your control arm (or harmful, or resource-intensive and ineffective treatment arms) than you need to. This latter point is much more of a problem in medicine than it is in other settings, as you can imagine. These considerations point to adaptive designs that balance exploration/exploitation, e.g. those based on multi-armed bandits. Currently working on some cool (in my opinion) variations of MABs that incorporate domain-specific knowledge, so I could talk about this all day!
Re:Successful machine learning models: lessons learned at Bo... by tixocloud on Monday October 07, 2019 @06:13PM (#21186545)
Would be really keen to hear you speak about the subject in greater detail actually. Love having to balance the practicalities of implementing a model in production and validating the outcome while not missing out on utility.
Re:Successful machine learning models: lessons learned at Bo... by joker3 on Monday October 07, 2019 @04:13PM (#21185453)
Do you know a good introduction to adaptive/sequential designs? I'm looking for something along the lines of a textbook aimed at a graduate level seminar.
Re:Successful machine learning models: lessons learned at Bo... by chudi on Tuesday October 08, 2019 @04:17PM (#21196707)
Re:Successful machine learning models: lessons learned at Bo... by Breza on Wednesday October 16, 2019 @01:30PM (#21272968)
Thanks!
Re:Successful machine learning models: lessons learned at Bo... by joker3 on Monday October 07, 2019 @03:04PM (#21184696)
I'm just going to present the absolute basics of the topic. It'll be a high-level overview of something along the lines of chapters 1-4 of https://www.amazon.com/Analysis-Experiments-Chapman-Statisti....
Re:Successful machine learning models: lessons learned at Bo... by jj12345 on Monday October 07, 2019 @03:41PM (#21185114)
Thanks for the reference!