This has been my experience in the US as well, but I'm not so sure that this paradigm extends outside the US.

I lived in Europe for a few years and didn't feel that same context as well - it wasn't almost assumed that white people were racist and anyone might be seen as racist regardless of their skin color or heritage.