is routinely done in countries in which variants of land value tax are implemented.
Retrofitting or tearing down a house in California is expensive. Particularly if it's an old house where there could be e.g. lead or asbestos. Clean lots (presuming it wasn't previously a laundromat) command a value from the baggage they clearly don't bring.
Lots of people want to design their own homes. For them, paying up for an empty lot is worth it. Put another way: those buying empty lots are developers or rich. With rates rising, the latter set marginal pricing in desirable places.
So no, there is no teardown required. But a teardown is desirable to some people. They bid up empty lots. Hence the deviance you notice, which is well documented in other places rich people like building designer homes. (Hamptons, Jackson Hole, et cetera.)
I mentioned the custom home premium upthread. It's not the cause of the deviance either.
Edit: As a weird example, subtracting what it would cost to build my California house from its current market value gives a value for the lot far lower than the actual market price of an empty lot in this location. This isn't just due to the depreciation of the construction (2012) but to market inefficiencies. It's actually the norm in our town. Only developers with economies of scale can really afford to build here, modulo a few wealthy families who are building their dream home without concern for ROI.