The reason Henry George thought it should be the only tax is because of the economic consequences. Since land is by and large fixed in quantity and a society's productive uses of land are largely dictated by the technology of the times then a Land Value Tax would not create inefficiencies in the market the way other taxes like tariffs or an income tax do.