It's not racist, but it introduces horrible historical baggage into a place it doesn't belong.

I don't mean to exaggerate... it's just words and we use all kinds of expressions that have roots in something objectionable.

Unfortunately people twist something from "mildly/moderately inappropriate" into "racist", which I agree is ridiculous.