Both are benefiting from the labour and value created by the rest of society, outsized to their contributions. Land receives its value due to all the infrastructure, people, businesses, and so on that are around the land, and everyone who owns land benefits from this as their land value increases. Owner-occupiers benefit just as much as landlords (they are, in fact, lords of their land too, but their tenant is themselves). They should return to society the value that society created for them, in the form of a land value tax.

This should of course be offset by a reduction in other taxes, but it is a supremely just and efficient tax and far preferred to most other taxes.