People are sensitive, impostor syndrome is extremely common. The people who work for you are always going to be worried about their job at some level. Don’t take advantage of your power to make a joke that is, at best, not funny and at worst incredibly anxiety-inducing.
A manager, like everyone else, should know who to make a joke to, and he will get closer to those employees.
Your job as a manager, with "sensitive" reportees is not to be a dick and make things uncomfortable for them, it's to help them do the work they're employed for.
It's like when a cop makes jokes about shooting you after pulling you over for expired tabs (happened when I was a teen).
Most of the Evil in This World Is Done by People with Good Intentions - https://quoteinvestigator.com/2019/12/19/intentions/
I would argue that breeding camaraderie requires empathy and understanding with other person as opposed to blindly projecting one’s sense of humor upon another.
This is getting strangely personal. If you are joking, well done on the subtlety!
People in positions of power should not take lightly their ability to ruin peoples' lives with strangers. Or anyone, really, but especially strangers.