Or people with autism spectrum disorders, or other issues (just being tired, say), who might find the nuance of your joke impossible to gauge?

Your job as a manager, with "sensitive" reportees is not to be a dick and make things uncomfortable for them, it's to help them do the work they're employed for.