I wonder if A/B testing is a bit like that. You're suddenly listening to every single issue encountered by users, but lack the wisdom to understand which issues to ignore. As a result, software changes constantly, users cannot really learn a GUI because it changes so frequently, and software teams feel like they're constantly improving things, however the software itself is not actually getting better when a the whole user's experience is taken into consideration.

We know how to optimize for things like the shortest distance between two points but we don't understand how to optimize Software Design or even GUI.

This is literally what "Design" means. If something needs to be designed it means there's no theory or notion behind it on what it means to be "optimal." So we "design" a "better" solution but we don't actually know if it's better. So we design another solution and the cycle continues.

A/B testing when done on a population that gives consistent answers should converge on a consistent solution. If the population gives different answers at different times then of course there will be churn.

I would say the methodology of A/B testing is indeed like machine learning, it's quite good and accurate. It's the data source that's the problem. If you have users that don't know what they want or behave differently and inconsistently then your conclusions reflect the data. Perhaps the data is accurate and there is no consistent conclusion, OR the data is inaccurate and you need to get it from a better source other then users telling you what they think is better.

Here are some dubious assumptions:

* There is an optimal answer.

* Human preferences are transitive (if I prefer A over B and B over C, then I prefer A over C).

* Human preferences are internally consistent.

* Human preferences are stable over time.

Apply these assumptions to finding an answer to the question of what you should make for dinner, and you'll quickly see there are problems.

> question of what you should make for dinner, and you'll

> quickly see there are problems.

And, to amplify your point a bit, there are problems with A/B testing for which of my children I should love the most, or what I find beautiful or interesting; or should I A/B test out whether enslaving others works out well for me?

I chose extreme examples, but the point is that there are many, many things in human experience that don't lend themselves to easy, simplistic rules. It's genuinly hard to work through a lot of a lot of the issues that people face in real life.

With that said, it's it's also important to work through and to understand objective data as best as possible. For clearly defined and nicely-behaving problems objective data is certainly the way to go. The problem is that a lot of the problems people actually face aren't so easy to understand, and aren't so well-behaved.

https://en.wikipedia.org/wiki/McNamara_fallacy

LBJ didn't want to "lose" Vietnam. LBJ remembered Joseph McCarthy who accused the Truman administration of losing China to communism.

JFK and Ngo Dinh Diem were murdered in 63, Gulf of Tonkin incident in 64, and full US armed intervention by 65. This was a rollercoaster ride that the US military didn't plan for and didn't want.

McNamara picked Westmoreland to lead the military effort. It was a disaster. Their lack of a coherent strategy was doomed from the start. The body counts and #bullets per enemy killed was akin to rearranging the deck chairs on a sinking Titanic. On top of their minds was the Korean War, when the US military operated at China's border and China intervened militarily. Hence, US military land operations was limited to South Vietnam. There were no plans to invade North Vietnam.

Nixon and Kissinger decided the end game was to breakup the USSR-China alliance. With China on the US' side, the USA no longer had a national interest in what happens to Vietnam. The US left Vietnam in 73 and the country was overrun two years later.

If you don't know how to properly construct experiments and interpret data it's honestly not much better than hiring an ornithomancer to spot omens in the skies. To be clear, this is something that even scientists with decades of both education and experience sometimes get wrong. If you're coming at it with an MBA I've got some concerns.

The entire problem here is that when people say "data-driven", what they actually mean is "model blind". If you make decisions based on a model, nobody will say it's data-driven, it doesn't matter how empirical the model is. Yet, if you rephrase the question as "Do model blind companies win?" the answer is obvious.

Running incoherent experiments testing for each little event you may think of is certainly better than walking at random. But as everybody knows for centuries, you use data to improve your model, and use your model to improve your craft. Jumping over the model part is an incredibly lousy use of data.

Data is useful for proving my decisions once I have made them, so what's most useful is having short feedback loops so can I catch bad decisions quickly, or reinforce good decisions.

I can't figure out what this means, but I have a feeling it's the exact opposite way you should be incorporating data into your decision making.

Prove (?), v. t.

1. To try or to ascertain by an experiment, or by a test or standard; to test; as, to prove the strength of gunpowder or of ordnance; to prove the contents of a vessel by a standard measure.

Thou hast proved mine heart. Ps. xvii. 3.

(Webster's 1913)

Which makes sense to me. Try something you think might work, then test it—is just plain old experimental science.

Generally, you can only generate data and do experiments based on your present state.

This sounds like post-hoc, anti-intellectual rationalization.

How do you place long-term bets without a model to measure expectations versus outcomes? How do you know what good intuition is? There is a ton of research on "gut" calls that demonstrates it's random.

>But data is not a substitute for good judgment

Good judgement requires data.

How is this anti-intellectual?

If you want, I can formalize as a game the problem of choosing business/product strategy in a competitive market with a continuous flow of imperfect information. I can then use ideas from controls to establish some upper bounds on what can be inferred from a continuous flow of information. I can then use that result to prove an impossibility result about the game. I can even tweak assumptions to get bounds on probability distributions which infer we'd be better off flipping a coin or whatever.

I'm not going to do the work, because intuition is almost always enough to identify these situations, but it's absolutely clear to me that results like this obviously exist and correspond to many real-world situations.

> Good judgement requires data.

It used to be that insisting on data-driven decision making was a hard pull. Now it's the opposite. Insisting on data where data cannot possibly provide enough signal to make a decision is the new form of anti-intellectualism. IMO.

I would say good judgement requires experience. Data may or may not be available and applicable, but its absence doesn't mean one can't exhibit good judgement.

Surely the data comes after the action, not prior to it?