I think you'll find that plenty of people are willing to not think so hard about such problems as long as it's in the course of a legal business. I'm sure it was also easy to find justification from commentators at the time when slavery was legal. You could just go about your slavery business and point at some guy's essay as to why you shouldn't feel guilty at all.