The wealth was not generated by the person doing anything but by all the activity occurring around them! Adam Smith despised landlords and wealth accumulation based on land-ownership for exactly this reason! They’re a parasitic drag on market economies that shifts capital from productive activities to unproductive ones (i.e. speculating on real estate).

“As soon as the land of any country has all become private property, the landlords, like all other men, love to reap where they never sowed, and demand a rent even for its natural produce.”