Black and White are not interchangeable. For your rule of thumb to make sense, you'd have to believe that Black and White people experience the US in the same way.
People will argue over the facts, naturally. That's as it should be. But if you are encouraged to criticize one race without penalty, whether the criticism is factually correct or not, but you aren't allowed to criticize another without being guilty of "racism", whether the criticism is factually correct or not, it's not about the facts. It's about who you can criticize and who you can't, regardless of your specific claims or evidence.
In other words, if you want to judge its nature, swap races. If the only thing that would happen if you swapped races is someone would say, "well then that wouldn't be factually correct", then it IS just about facts. But if instead, those in power would rage about racism and punish the "offender", they are not merely disputing facts. Something else is happening.
Suppose you criticize someone for being slow at reading. It's not the nicest thing to say, but it's run of the mill as criticisms go. But then suppose you criticize someone for being slow at reading who has dyslexia. That would be considered extremely insensitive and rude, even if it's factually correct.
Suppose you say "I screwed up". Pretty unremarkable; self-deprecation is common in our culture, and people won't consider it a sign you're being too hard on yourself, unless the mistake you made was extremely minor or harmless. But then suppose you say "You screwed up". That would be considered far too aggressive and direct for most situations.
Of course it matters who you're criticizing.
Even if it is Americans making these statements in an American context, the Internet and the media broadcasts them to the rest of the planet.
You have to think, not just about whether they make sense in an American context, but also whether they make sense in the non-American contexts in which they are being received.
Your other comment about the Australian experience was interesting and it's useful when comparing the US to the rest of the world. However, it feels like you're trying to say I'm off topic because I'm discussing an American document and an American reaction by using American history.
This American talk about "whiteness" doesn't just get posted in an American museum. It gets posted on Twitter etc and then people on the other side of the planet read it. And the people on the other side of the planet may understand it very differently. But Americans never seem to think about that. (And I'm not saying the concept of "whiteness" might not have some application or usefulness in other countries, including Australia-but it certainly doesn't have an identical application.)
Or, maybe one is a non-US-based employee of a US-based multinational company, and the company leadership starts promoting all this talk to the entire company, without seeming to ever stop to think about how much sense it makes in a non US-context. (But would any non-American employees dare raise the question of how US-centric the biases of its US-based management are?)
So, my relatives want to know, are the baizuo (white leftists) claiming that in China, nothing white people say about or do to Chinese or blacks can be racist? Or are they claiming that they are actually "in charge" of us in China?
So whatever that infographic is, if the people who wrote it would consider it a racist attack if someone else did it and used "black" instead of "white", it tells you how they really intend it, regardless of how they might edit the definition of "racism".