Exactly. The biggest problem with quantifying efficiency in particular is that you have to define the inputs and outputs. Your biases are embedded in that definition rather than in the calculation per se. You can make some pretty awful processes "efficient" if you assign zero value to some of the inputs, and conversely some pretty great processes "inefficient" if you assign zero value to some of the outputs.
I'm halfway convinced that time-motion studies are some kind of demonic invention for exactly this reason. We can get 3% more productivity out of an assembly line, just by doubling the accident rate and giving every worker RSI!
Goodhart's Law is dangerous even when we're making management and business decisions in good faith, but when ideas like "long term injuries are irrelevant" are implicitly added it becomes monstrous.
Wouldn't it make his result even stronger? If the thesis fails to hold even in better conditions (like when there is a well-defined expected value), it should also fail to hold in worse conditions.
That’s really not true at all — key decisions like whether to relieve a pitcher are not made within the context of a game. At a higher level, decisions to trade players are made that will affect outcome as well.
As someone who manages projects, there are parallels from a context point of view to be drawn to baseball. Each project is a measure of success or failure, but my effectiveness isn’t defined by individual projects. I may choose to do a project less efficiently to control cash flow or prioritize something else.
Small aside but some including me would have a problem with even using sports as a proxy. There's a school of thought in soccer that maintains that the best match is a 0-0 draw [1][2][3]. While the data reflects a completely unproductive match, a 0-0 draw means that both sides performed admirably in defending and in countering the efforts of the other side. There is very detailed stats in soccer too now, but, that doesn't weaken the point.
[1] http://lastwordonfootball.com/2017/09/09/five-famous-0-0-dra...
[2] https://www.theguardian.com/football/2007/aug/14/newsstory.s...
[3] http://worldsoccertalk.com/2010/06/22/explaining-a-draw-to-t...
At the risk of sounding like an old fogey, I believe a similar statement can be made for stats-utopian circles and their belief in ability to quantify every thing.
As the post says:
> One nice thing about sports is that they often have detailed play-by-play data and well-defined win criteria which lets us tell, on average, what the expected value of a decision is.
So, using sports as a proxy is not the same as management decisions where there are no well-defined criteria or expected value.