That reasoning works on land. (Hence a land value tax can't be passed on to tenants.)

But it doesn't work on the houses on top of the land. Because we can build more or less of them. Or higher.