Since “harmful” isn’t generally used to describe people, it’s not generally an issue. “Toxic”, on the other hand, I see applied to people all the time.