I understand it to be harmful, but in a way that contaminates the environment around it. A bad colleague can be a dick, in isolation, whereas a toxic colleague would be someone whose behaviours poisons their surrounds and brings out bad behavior in other people in their team.

I am non-contact with a toxic parent because I know it brings out worse behaviour in myself, and I want to stop that cycle of passive-aggressiveness and judgement towards people I do care about.