That's true. That's why there's also the umbrella term, "bigoted", when referring to bias stemming from ignorant prejudice. The watering down of "racist" to equal all forms of bigot is indeed a problem, and you have brought up a valid concern.