I recommend you read "The New Jim Crow: Mass Incarceration in the Age of Colorblindness". The relevant portion of which I'm thinking relates to the historical affinity between disenfranchised blacks and whites in the early 1800's, and how a race-based conflict was stoked by those fearing a coalition of these two groups. There is an interesting historical context for the point you're making.