Data doesn't know what colour you are by looking at you. It only knows what colour you are when you or someone you know specifies that.
For example - your car insurance company doesn't know that you're black... unless you include that on the form. And it doesn't matter. Black people are no more or less prone to having an accident than Hispanics or Chinese or Whites. It's irrelevant for the purpose of car insurance. But insurance companies will ask whatever they can get away with and claim to use it to provide discounts based on statistics. They then often package up this information and sell it to other companies who now know you are black, just because you provided that information when it seemed advantageous to do so. But they will equally use that information against you if they have the opportunity to deny your claim.
I might look at you and know immediately that you're black, but what if I don't know you? What if I've never seen your picture? The only thing I can glean from your data is what data there is to sift through.
So just because something may be visible in person, doesn't make it less important when it comes to privacy.
Even the most innocent of data can be misconstrued and turned against you.
That you live near or socialise with possible terrorists and criminals doesn't make you guilty. I've had very close friends amaze me when I find they're to be divorced as they always seemed a happy couple. I guess friends don't tell me everything. That includes not sharing their plans for terrorism, bank raids and coups.
So McCarthy era paranoia but with an awful lot of data to draw unfair inferences from or perpetuate society's current biases.