"afford" is just a way to rephrase "exploit" - due to the fundamentally unfair wealth distribution

you can "afford" a gardener as a retired middle manager only because a lot of poeple in that society are being enslaved

What a great idea! Let's retire there and have slaves!