You're right that you're not the only one - this way of thinking is common, it's known as "fragile masculinity". As you've found, if you worry too much about people with very fragile masculinity, you won't be able to enjoy things you want to enjoy. I think it makes sense to just not worry about these people.