The imperative for the government is to reduce the deadweight loss of under-improved land. Whether or not the owner chooses to improve should be their choice, but there is an opportunity cost to not improving.

Absent a LVT, that opportunity cost is mostly born by the people who don't own the land, i.e. the coffee shop that _could_ have been there instead of whatever lower value use is there now. With a LVT, that opportunity cost falls more on the person who owns the land, i.e. the owner wants to keep their SFH in the urban neighborhood, but now pays a higher tax which reflects that under-improvement.