Whether you work or not doesn't cost me anything. But if you're hoarding a natural resource and playing games with the supply then you're costing me in increased rents; not just for me but for everyone around me. Your actions are creating a negative economic externality that I am not being compensated for. Extracting rents is a form of theft from society; a diffuse and difficult to trace theft but real, damaging and if allowed to continue unchecked a threat to the stability of society.