News for hackers. Stuff that matters.
Re:U.S. users are leaving Facebook by the millions, Edison R... by arethuza on Thursday March 07, 2019 @05:22AM (#19326938)
Isn't A/B testing pretty much going to behave like a steepest ascent hill climbing? At each micro decision point you take what looks like the 'best' option but that means you can get stuck in local maxima?
Re:U.S. users are leaving Facebook by the millions, Edison R... by Matheus28 on Thursday March 07, 2019 @03:48PM (#19332210)
Yes, but it gets more complicated because there's a time factor too.

If you're A/B testing each change for 2 weeks, but the negative impacts of it only happen after a few months (like what the parent post mentioned [1]), then while you're in a local maxima right now, it'll slowly sink, along with all your neighborhood of choices.

You can think of it as a function that returns the current value and another function that you have to use for the next time step. Sorta like f(a, b, c, ...) = (y, \a_2 b_2 c_2 ... -> ...). Steepest ascent hill climbing doesn't work well for finding good long term local maxima, since you don't know how long it takes until it stabilizes (or if it ever does). The best you can do is guess it'll stabilize in X amount of time, but if X is too small, you might end up stuck in a really bad local maxima.

[1] https://news.ycombinator.com/item?id=19325816

Re:U.S. users are leaving Facebook by the millions, Edison R... by jmilloy on Thursday March 07, 2019 @08:55AM (#19327978)
I think the bigger problem is that A/B only measures certain things and following it blindly can have you descending in important ways that are hard to measure.
Re:U.S. users are leaving Facebook by the millions, Edison R... by Yhippa on Thursday March 07, 2019 @08:47AM (#19327933)
This is interesting but I don't think I fully understand. Do you mind dumbing it down for me?
Re:U.S. users are leaving Facebook by the millions, Edison R... by JorgeGT on Thursday March 07, 2019 @02:25PM (#19331260)
Typical example: you arrive at a crossroads A where both ways work for you, so you choose the one with less traffic. Then at crossroad B you do the same, and then at C, and finally you arrive at destination D.

However, it turns out the heavy traffic at the other A branch was just for a few miles and then it was actually empty after that --- you took optimum local decisions at each step but since you weren't able to look at the big picture, you didn't actually choose the globally optimal route.

As others have pointed, this is related to the mathematical concepts of local and global maxima: sometimes your optimization algorithm happily stops when it finds a local maximum, ignoring the much better global maximum because it didn't actually traversed the whole search domain.

Re:U.S. users are leaving Facebook by the millions, Edison R... by girmad on Thursday March 07, 2019 @08:58AM (#19328003)
Global vs local maximum. https://en.wikipedia.org/wiki/Maxima_and_minima

Related: what's the best for one part of the system, may not be the best for the entire system.

Re:U.S. users are leaving Facebook by the millions, Edison R... by qubex on Thursday March 07, 2019 @06:08AM (#19327143)
Correct.