but the value (including the location) only exists if the eventual goal of adding improvement to the land is possible (or is doable). If i stipulate that the land is forever unused (e.g., i contaminated it with poison which cannot be removed), then the land would become worthless, regardless of the location!

They care about what a currently empty lot (perhaps with necessary building permissions) fetches on the market.