In my view, the only problem with slavery is that it's involuntary. It's important to make a distinction between voluntary servitude and involuntary servitude (real slavery). To prevent someone from voluntary servitude would require their involuntary servitude to whomever is preventing them.

To put it another way: If person A is forcefully preventing me from voluntarily serving person B, that makes me person A's involuntary slave.