News for hackers. Stuff that matters.
Re:Goodhart's Law by rossdavidh on Friday September 17, 2021 @01:02PM (#28567871)
In the context of manufacturing, W.E.Deming said something similar: "that which gets measured, gets improved". His conclusion from this was a little different than McNamara's, though. Since you will inevitably want to track your progress, make sure you track as many things as possible, because anything which is not tracked will get sacrificed to that which is. Up to a point, it's true.

One issue is that some things, like vulnerability to supply chain disruptions, are intrinsically harder to track because they are based on rare occurrences. Thus, they will tend to get sacrificed in favor of measures which are more frequent, leading to an emphasis on short-term strategies.

Re:Goodhart's Law by CalChris on Friday September 17, 2021 @01:40PM (#28568444)
Actually, Deming said much the opposite:

  It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.
p. 26 of The New Economics for Industry, Government, Education

It wasn't Drucker either.

https://medium.com/centre-for-public-impact/what-gets-measur...

This whole mindless must measure, must measure mentality has been criticized since the 50s. Measurement is a tool. There are many tools.

Re:Goodhart's Law by mumblemumble on Friday September 17, 2021 @04:24PM (#28570491)
I would argue that it's worthwhile to measure as much as you can, insofar as it facilitates orderly decisionmaking processes.

The problem is that people tend to think that all measurement is necessarily quantitative. I think that this might be a version of the streetlight effect? Quantitative measurements tend to be much easier to collect and analyze than qualitative measurements. Oftentimes you can let it all run on autopilot, whereas doing good qualitative work always requires concentration, effort, and expertise.

Re:Goodhart's Law by dragonwriter on Friday September 17, 2021 @08:05PM (#28572258)
> I would argue that it's worthwhile to measure as much as you can, insofar as it facilitates orderly decisionmaking processes.

That would be true, if measurement was free. It never is, and it often is quite costly.

Re:Goodhart's Law by dr_dshiv on Sunday September 19, 2021 @07:55AM (#28583623)
What is qualitative measurement? As soon as it is quantitative, it isn't qualitative anymore, is it?
Re:Goodhart's Law by mumblemumble on Sunday September 19, 2021 @06:52PM (#28588686)
Re:Goodhart's Law by rossdavidh on Friday September 17, 2021 @03:09PM (#28569688)
In my experience, if it isn't measured, then it is assumed that the policy (whatever it is) is working. If you don't measure, you can't be surprised by "wow, it didn't work like we thought". Therefore, mistakes don't get recognized or corrected. There are many tools, but measurement is one of the only ones that brings unexpected bad news to the user, and that is invaluable.
Re:Goodhart's Law by mhink on Friday September 17, 2021 @01:40PM (#28568455)
> One issue is that some things, like vulnerability to supply chain disruptions, are intrinsically harder to track because they are based on rare occurrences. Thus, they will tend to get sacrificed in favor of measures which are more frequent, leading to an emphasis on short-term strategies.

I suppose this is part of why "chaos engineering" has gained popularity- introducing artificial disruptions at a known rate makes it easier to quantify the impact of otherwise-unusual events.

Re:Goodhart's Law by rossdavidh on Friday September 17, 2021 @05:30PM (#28571129)
Ooh, good point! Another example is auditing, where you substitute regular, frequent disruptions (being audited is disruptive to normal operations) for infrequent, less predictable, bigger disruptions.
Re:Goodhart's Law by dragonwriter on Friday September 17, 2021 @08:04PM (#28572250)
> from this was a little different than McNamara's, though. Since you will inevitably want to track your progress, make sure you track as many things as possible, because anything which is not tracked will get sacrificed to that which is.

That wasn't Deming’s position, Deming was pretty strong on the idea that there are measurable things that it is not cost effective to measure.

A big contribution of Deming’s here is the importance of understanding uncertainty in measurements relations between them to understand the degree of control.

Re:Goodhart's Law by equality_1138 on Friday September 17, 2021 @11:13PM (#28573313)
Fredrick Taylor was the one who developed so-called management science and obsessive measurement, paving the way for Deming to popularize a more balanced or holistic approach, ie. akin to actual science.