Isn't A/B testing pretty much going to behave like a steepest ascent hill climbing? At each micro decision point you take what looks like the 'best' option but that means you can get stuck in local maxima?

Yes, but it gets more complicated because there's a time factor too.

If you're A/B testing each change for 2 weeks, but the negative impacts of it only happen after a few months (like what the parent post mentioned [1]), then while you're in a local maxima right now, it'll slowly sink, along with all your neighborhood of choices.

You can think of it as a function that returns the current value and another function that you have to use for the next time step. Sorta like f(a, b, c, ...) = (y, \a_2 b_2 c_2 ... -> ...). Steepest ascent hill climbing doesn't work well for finding good long term local maxima, since you don't know how long it takes until it stabilizes (or if it ever does). The best you can do is guess it'll stabilize in X amount of time, but if X is too small, you might end up stuck in a really bad local maxima.

[1] https://news.ycombinator.com/item?id=19325816

I think the bigger problem is that A/B only measures certain things and following it blindly can have you descending in important ways that are hard to measure.

This is interesting but I don't think I fully understand. Do you mind dumbing it down for me?

Typical example: you arrive at a crossroads A where both ways work for you, so you choose the one with less traffic. Then at crossroad B you do the same, and then at C, and finally you arrive at destination D.

However, it turns out the heavy traffic at the other A branch was just for a few miles and then it was actually empty after that --- you took optimum local decisions at each step but since you weren't able to look at the big picture, you didn't actually choose the globally optimal route.

As others have pointed, this is related to the mathematical concepts of local and global maxima: sometimes your optimization algorithm happily stops when it finds a local maximum, ignoring the much better global maximum because it didn't actually traversed the whole search domain.

Global vs local maximum. https://en.wikipedia.org/wiki/Maxima_and_minima

Related: what's the best for one part of the system, may not be the best for the entire system.

Correct.

Exactly this. Death by a thousand paper cuts, is another.

I wonder if they even do their A/B testing right. In my experience (none at facebook) accidental p-hacking is rampant in the tech industry, with trials being cut short or prolonged by the tester who's staring at a graph of the results in real time.

If you have a change that improves your metric initially but damages it in the longer term, and your A/B test only detects the short initial effect, you can execute your A/B test perfectly and still get the wrong result.

Interestingly that's exactly how algorithms make things go viral. They pick up on things that get a quick reaction with complete disregard for what happens in the long term. Jaron Lanier explains this in his talks [0].

So features on social media are decided based on short term gains and posts on social media are promoted like that too. It's like an entire industry forgot their parents warnings about thinking about the future.

[0] https://www.youtube.com/watch?v=kc_Jq42Og7Q

Or Hacker News is wrong.

You are maybe being downvoted as you stated it so bluntly, but I'm not sure your point is without merit.

The HN audience do not necessarily represent the mean/mode user, and Facebook are in a numbers game really.

I agree with most of the sentiment above - I wish I could filter posts that are just attachments, 3rd party junk, and tune the algorithm to show me posts from a core set of friends, but I also recognise I don't use FB like a most people, I imagine.