[1] https://en.wikipedia.org/wiki/Dirty,_dangerous_and_demeaning
The problem from my pov is people get defensive when someone says "Japan is great. it's so much safer than the US it makes the US feel like a 3rd world country" which is true IMO and it's said to emphasize how the US (and many other countries) take for granted the amount of crime. And sadly instead of getting the intended message which is "the level of crime you live with in the USA is something to be fixed" instead they resort to pointing out the bad parts of Japan so they can ignore the harsh conclusion.
I guess one can say that generalizing something so dire about an entire population would also, perhaps unwittingly, be racist.
Would it be racist to say that slavery in the pre-Civil War United States south was a societal issue? Or to say that racism in Nazi Germany against Jews was a societal issue? I don't think so. (I give these example to demonstrate that there is some line a society can cross where you'd hopefully agree it's an issue.)
I don't know whether the claims are true or not about Japan, but I don't think it's inherently racist against the Japanese to claim that xenophobia or racism are societal issue there.
To believe that all such accusations are inherently racist is to believe in a form of moral relativism in which no society can ever be justifiably called racist.
It is both a place worth visiting and also full of what we would call racism. The overly-politically correct crowd in America would be (and often are) shocked at what passes for acceptable race-related behavior in Japan.