It's not some moral relativism that depends on the context. Right and wrong are absolutes, and can be objectively appraised. And here's the metric: Suffering. Right actions are those actions that lead to a reduction in the suffering of living things. Both in others and in ourselves. Wrong actions are those actions that lead to an increase in suffering in ourselves or others.
This is how we can appraise our actions. This is totally off topic, and I'm definitely showing my Buddhist beliefs, but I think the attitude shown by your last sentence is held by a lot of people in society. And I think it is also muddled thinking. This kind of 'moral relativity' is actually a slippery slope and has been used to justify all manner of human atrocities through history.
And a bit of a tangent, I don't disagree with your last statement in the specific. But in principle, that's a very dangerous statement to make: if you're living within the moral constraint of a society, there really should NOT be any criteria that define a human to be a better or worse person.