I am not saying "you can't say that". I am saying that the context of who is speaking matters quite a bit. It may not seem 'fair' that when speaking on how hurtful a statement is to a particular group, whether you are actually a member of that particular group is highly relevant, but that perception of unfairness is itself an aspect of privilege.

Basically, if you make a joke about a group you aren't a member of, don't expect to be able to tell someone who is a member that it's 'just a joke', and that goes double if you're a member of the dominant group (whichever that happens to be in context).

Personally, I hit almost all of the privilege checkboxes with the (very mild) exception of religion/ethnicity. Guess what? If you aren't Jewish, and you tell me a Jewish joke, and I take offense (and I might even if it is a joke I have used myself), and I then call you out on it, 'lighten up' just isn't an appropriate response, regardless of the fact that you may not have privilege in some other area.