That's like saying we should not have an absolute right to be more intelligent than the average person since such a right is perfectly consistent with a society where one person is infinitely more intelligent than everyone else.
Material equality is not a right, and the pursuit of a society with material equality does not grant any party a right to imprison someone for their refusal to hand over a share of the currency they receive in private trade.
There is a fundamental moral difference between using violent threats and physical confinement from using offers of payment to compel work.
Taking a job because the alternative is to starve is still a consensual interaction. We need to work to survive. We can go out into the wilderness, and live off the land, which would let us survive at subsistence levels, or we can go into civilization and work within an employment contract that gives us a much higher standard of living.
The freedom to choose ensures that generally speaking, people in a market economy are choosing one of the better options available to them, rather than one of the poorer ones. It also ensures that when people DO interact with others, it is because it is generally mutually beneficial. This basic dynamic leads to an emergent property of actions carried out in markets generally being a net-value gain for the economy as a whole.
If someone points a gun at you and tells you to work, your choice is obedience or death.
If someone hoards all the food and tells you to work for it, your choice is obedience or death.
In what way are these meaningfully different?
>We can go out into the wilderness, and live off the land
No you can't, because the tyrant owns all the land and calls the police if you try to take any of their food.
Such a society would be not be just and nobody wants to live in it. The general idea to prevent this kind of barbarism is that additional fairness criteria are needed that ensure that everybody can live in dignity, and only if your wealth does not violate these principles should you have a right to it. Fair resource distribution principally conflicts with the right to one's own property and unfortunately out of the box also with the Pareto efficiency of transfers.