It's partly context, but it's also that you don't get to choose whether a particular word in a particular context is offensive to another person.

What is socially acceptable is decided by society, not by any particular individual. Mind you, that doesn't prevent us from being more careful around people we know to sensitive to certain forms of address, but in a public forum that's not really possible because you don't know who you're talking to.

Yes, but we can mock the offense as inferred rather than implied.

Would you mind me calling you "boy" in general? How about a cutesy corruption of the word "boy"?

The etymology of the word is irrelevant to how it's used conversationally.

Words carry history. I can think of words that you might use in conversation with various groups without meaning to project any negative overtones whatsoever that would nonetheless be extraordinarily badly received; you could explain to the paramedic as he stems the bleeding that you didn't actually mean to cause offense.

FWIW, I've got three daughters, all adults now. We've talked about the evolution of the 'casual sex nomitive' and their consensus was that as a group it was ok to call everyone guys like "Hey guys, let's go to the beach!" but in the singular it they preferred the use of 'gal'. Further the use of guy/gal was preferred in casual speech over man/woman.

I suspect at some point every word may be pejorative in some way.

Well, I hope you never call anyone "guy" then either, since it's derived from the name of a religious fanatic and (attempted) terrorist.

No, it's derived from an actual name. http://www.behindthename.com/name/guy

Just because there was someone famous with a name doesn't mean that name was derived from that famous person. It took on new meaning in the UK after that famous person became famous, but he didn't invent it. While we're here, he wasn't a failed terrorist. He was a failed assassin.