I know. My point is not about IQ. My point is that people find IQ to be a useful measure of things, in certain situations. We can use something else, like the big 5 personality inventory, if you like. The point is just that subjective characteristics of a person are things that we quantify and sometimes care about measuring.

And the times when we find ourselves wanting to quantify and measure them in order to make decisions that impact people’s lives them are precisely when we need to be extremely careful, because they are subjective traits.

Of course we should be careful. But we should also keep in mind what the baseline is. The baseline is subjective evaluations by other humans. If we don't build models, it's not like we're going to live in a world where nobody's subjective personality traits are evaluated. They still will be. They'll just be evaluated by other humans in an opaque, un-monitorable way. Using models at least makes it transparent, and something that we can iterate on to remove bias. There are no effective ways i'm aware of to de-bias humans.

If the human baseline is bad, that doesn't imply that it's better to use computers to do the same task much faster while obfuscating accountability.

You have it exactly backwards though. Its humans that obstruct accountability, because they are opaque and inconsistent. ML is consistent and inspectable. It cannot lie to you about its motivations. It is infinitely more accountable than any human system ever will be.

Is amount of money in someone's bank account a subjective trait? IQ has been shown to correlate with income.

That correlation could be very easy to explain by a confounding variable.

confounding variables can't be quickly and cheaply measured, that's what IQ is - measuring all the confounding variables.