Where we have measurements for pretty much everything but people don't know measurements are proxies and not always aligned with goals. Like how a ruler doesn't measure meters, but meters according to the ruler and only at the ruler's precision level. I can totally get how people who don't work with these tools don't understand the nuances, but it confuses me with experts. Isn't expertise, by definition, contingent upon understanding nuance?
It seems that the more metrics we have available to us, the less we care about understanding how those metrics work and what their limitations are. That they just become black boxes.
I mean there seems to be a very common belief that you can measure visual image quality by measuring some difference between an intermediate representation of a classification model. Or a belief that entropy measures quality of speech.
I'm really concerned that Goodhart's Law might be one of the Great Filters.
My theory is the increase in available information is overwhelming everyone's cognitive abilities, and so jumping to reductive conclusions is just a natural defense mechanism, leading to increased polarization and reduced listening/tolerance skills. Certainly software engineers have an above-average tolerance for nuance, but even we have to pick our battles or risk drowning in the flood.