I was going to say something about how I occasionally use the phrase "white privilege" and I agree with you, but I think I actually just tend to use "privilege". You're right, in different times and places there's a lot of change in who is privileged. And it's definitely not a two-level system. I don't think the word "privilege" necessarily implies it, but it's used like that a lot, and I think that's unfortunate, and it's definitely a simplistic view of the world.

For instance, Irish-Americans did not have a particularly great time in the US in the mid-19th century by any means, but they had a much better time than African-Americans. They faced serious discrimination, but they rarely were assumed to be runaway slaves.

I think it's useful to acknowledge that even today, some groups of people have basic privileges like being assumed not to be a thief that everyone ought to have, and we can acknowledge that without reducing it to a simple white-nonwhite line.