I don't think I'd make this "Joke" in the first place, but if I was a 3rd-person observer to this scenario my honest first reaction would be (1) that's a weak attempt at humour, quickly followed by (2) that person is really sensitive.

I suspect the manager would interpret that sort of feedback as #2 as well