And why should some people benefit from progress, while others are left to live in the wilderness? (Considering only few people actually contribute to progress.)
Because progress is the net result, directly or indirectly, of the work of everyone living in that society. When you pay taxes, when you buy things, when you select which products to buy, you're contributing.
Rubbish. Nothing stops you from wandering off into the wilderness, except for the fact that it's a crap way to live. You just feel entitled to the benefits of modern civilisation, without the costs.