This is why it is sometimes referred to as "location value" rather than land value. The dirt is approximately worthless, but the location is worth a lot due to the surroundings (usually, because of human development of the surroundings).

Advocates of a land value tax therefore often call it a location value tax, or a site value tax, because it is the location and the society around the location that has value, not the dirt itself.