As for why we single out people who hate others because of skin colour, but not because of say hobbies: A large number of people have been murdered (or beaten, or denied employment, or denied housing, or etc etc) because they are black. I don't think I can find more than a handful of philatelists who were murdered because they collect stamps.
I think the touchiness around discrimination based on skin colour is understandable and based on plenty of evidence.
Yes of course, a lot of people aren’t killed for collecting stamps. But I don’t think many people today are killed specifically for being black or gay either. It does happen, yes. But I believe the overwhelming majority of people just carry on with life. We should all remember that whites vs. colored fountains were deemed lawful by the government.
I truly believe most people today just want to go to work, do their job, and get paid regardless of the gender or color of the people around them. And honestly, what more would you expect? We live in the country with the greatest amount of diversity you can’t escape. Americans think Americans are the least tolerant and my travels to other countries and interactions with different people shows me the exact opposite.
Edit: I say this as someone who grew up in the South. I lived around racists but most of that stays in their head.
Concerning your last point - I'd guess they're met with hostility and denial because they legitimately don't believe the issue is as widespread as many are led to believe.