It's a derogatory comment focused on one gender. The problem for me is that similar comments focused at women are often (and in my opinion correctly) viewed as sexist