Let's talk about the word "cracker". In the US, it's a derogatory term for white people that was coined by black people. The etymology is that a "cracker" was a white man who was hired by the plantation owner to crack the whip at the black slaves. Even in white society, these "crackers" were pretty much what we'd call "white trash"--they had effectively zero social status, received no largesse, and lived in poverty themselves. Literally the only thing they had going for them was that they were white, which meant they were the ones cracking the whips and not having the whips cracked at them. Which only made them all the more eager to crack the whips and do the dirty work of oppressing black people. Being white was literally the only thing these people had to be "proud" of. And so, the most evil people weren't necessarily the plantation owners (who were evil in a detached, hypocritical way), but rather the crackers (who were evil in a directly hateful, sadistic sense). A plantation owner could come around to the idea that slavery was wrong, as Washington and Jefferson did, and merely live in a state of complicated hypocrisy. A cracker is a pure mass of utter racial animus.

The same dynamic remained after the abolition of slavery, and it's the same dynamic that you describe.

Privilege is basically the logical inverse of discrimination. If driving while black means getting pulled over more (discrimination), then driving while white means not getting pulled over more (privilege).

Hope this clarifies how even poor whites are "privileged".