Quoting wikipedia https://en.wikipedia.org/wiki/Land_value_tax
> A land value tax (LVT) is a levy on the value of land without regard to buildings, personal property and other improvements upon it.
So if you were to own the lan under the Shamrock Pub in 501 E 8th St, Boston, MA 02127, United States you would pay taxes on that piece of land as if the Shamrock Pub was not there but obviously still considering the city around it.
Georgism asks you to not consider the improvements UPON your land, not to ignore those AROUND it.
The question of what that value is is quite easy to answer, it is the price of a similar empty lot in the same area, with the same developments around it.
When left as swampland, Back Bay is essentially valueless, both in aggregate and lot-by-lot. Someone in the past came along and improved that condition to the current relatively firm earth.