I'm a westerner, I lived in Japan for a year. I'd say it's both those things. The racism westerners experience is very different & much better than the 3K's type racism [1] other foreigners are subject to in Japan.

[1] https://en.wikipedia.org/wiki/Dirty,_dangerous_and_demeaning