I agree with your conclusion (I think we need to be very careful to edit language, lest we forget the lessons of the past) but from the list you gave none of them have a negative connotation - except perhaps black swan, but it's a reference more to rarity. Black hat has an immediate negative connotation.