The rules governing social systems built to obscure the jungle (e.g., political, legal, and penal systems) can always be trumped by that which they were chosen to tame. This is the unfortunate reality of our wetware.
> the thing people actually care about is the risk of bad outcome, not the actual data itself
"Bad" is subjective, no?
Is it good or bad if a father learns that his teenage son is not his own?
I feel like that would depend on the person. If the father wanted to know and the son didn't, that would be good for one and bad for the other, and vice versa.
Regardless of the outcome, I’m entitled to my privacy and my privacy is important to me simply for the sake of my autonomy.
Simply having my privacy violated is a bad outcome in and of itself.
Relinquishing privacy in order to participate in modern society in a meaningful way is one of the great frustrations of modern times… for some.
It's more than that. Privacy affects the psychological context for daily activity and alters behavior, sometimes subtly, sometimes overty. If a person knows they have no privacy, they will go about life completely differently, and they will think about life completely differently.
Privacy is a freedom-of-thought, freedom-of-action, and also freedom-from-anxiety consideration.
Any answer to why do you care about hiding this information? can all be boiled down to the fear that "[person or group] might use [private data item] to create [bad outcome] for me."
So the thing people actually care about is the risk of bad outcome, not the actual data itself.
If your theory is correct, then the focus should be on the prevention of asymmetric power imbalances in societal transactions that can even create [bad outcome].