conductrics

We needed a principled approach to help find heterogenous treatment effects and to discover if A/B Tests with more than 2 arms were potentially interacting with one another. Most approaches seem to just use single t-tests with multiple comparison adjustments, but this approach just became too unwieldly at scale. Anyone else use the F-test with nested regression? Or find some other useful approach beyond collections of individual A/B tests?

A simple Zero analysis Multi-Armed Bandit Reproposing Pearson-Neyman sample size calculations to create simple bandits with no need for dedicated software. Obviously not optimal but often a very good option.

Are you suggesting that revenues fell after bailing on SMB. That is hard to imagine, but I guess possible. Given that the core of the industry is about asking counterfactual questions, I would think the appropriate question would be 'would they be more valuable now if they had not gone to the enterprise - would they even have the deal they did wind up getting?'rather than are they more valuable now then they were 4 years ago. Hard to know, but my guess is that they wouldn't. A simple web editor with a random number generator isn't going to be of interest to anyone looking to buy. The larger problem is that statistical inference is hard - it just is. And, unlike analytics, where you are just placing sensors into an existing system, here you need to also place actuators, so implementation is much more complicated. That means that at any scale, both the marketing team, and the dev/IT teams need to be involved for anyone to get value and not wind up breaking systems all the time. Software like theirs, and ours, isn't magic, and without good editorial and hard work by the client, the entire exercise is more statistical theater than science. And that material fact was always in conflict with the rhetoric that they make AB Testing easy for everyone. It lends itself to a particular type of solutionism that VCs and the larger industry are prone to be seduced by. Self service to SMB clients might be a profitable biz, but perhaps not enough to service such a large amount of venture funding. FWIW VWO also tries compete at the enterprise.

When thinking about what type of approach is best, first think about the nature of the problem. First is it a real optimization problem, IOW are you more concerned with learning an optimal controller for your marketing application? If so then ask: 1) Is the problem/information perishable - for example Perishable: picking headlines for News articles; Not Perishable: Site redesign. If Perishable then Bandit might give you real returns. 2) Complexity: Are you using covariates (contextual bandits, Reinforcement learning with function approximation) or not. If you are, then you might want your targeting model to serve up best the predicted options in subspaces (frequent user types) that it has more experiences in and for it to explore more in less frequently visited areas (less common user types). 3) Scale/Automation: You have tons of transactional decision problems, and it just doesn't scale to have people running many AB Tests.

Often it is a mix - you might use a bandit approach with your predictive targeting, but you also should A/B tests the impact of your targeting model approach vs a current default and/or a random draw. see slides 59-65: http://www.slideshare.net/mgershoff/predictive-analytics-bro...

For a quick bandit overview check out: http://www.slideshare.net/mgershoff/conductrics-bandit-basic...

I read GEB as my subway book, back in 1999 (during a work stint in Paris), during my daily commute to work. Years later, when I was looking for a change, I started thinking about GEB. Even though I had no CS background,I decided to apply to a few AI programs and wound up going to the University of Edinburgh for an Msc. in AI in '05. Now I am a co-founder of software start up that applies reinforcement learning, a method from ai -lower case ;-) to conversion optimization. Obviously, I can't say if you will find it a worthwhile read, but I do look back on it as a significant influence on a major pivot in my life.

I think what makes Hinton surprising is that he has a long established academic lab and so many current top researchers went through that lab. Yann LeCun (ANNs/Deep Learning), Chris Williams, (GPs), Carl Rasmussen (GPs), Peter Dayan (NeuroScince and TD-Learning), Sam Roweis (RIP). As you note, industrial research labs (with Nobel prize winning researchers) have been around at IBM, NEC, and ATT Bell etc. One thing that I think about, is what happens to the quality of research as top folks who have an established record of producing new researchers are pulled from that role? Also not sure about startups having anything to do with with making technology real. Is Google still a startup?