Society is a collection of individuals, including many who opposed slavery (not all of whom were slaves).

It's turtles all the way down.