Sounds like there would be plenty of edge cases in this kind of thing too.

For example, apparently not all the slave owners were white:

https://www.encyclopedia.com/humanities/applied-and-social-s...

That's likely to complicate the heck out of things.