I generally agree with the principal you present but you have to acknowledge that the government (and therefore, theoretically, tax payers) gain value through the stadiums presence in the community. More directly through sales tax and tourism but possibly other less direct means like property values and property taxes.

Furthermore, if that's what people want to be done with their tax money, that's their business. If enough people felt the way you do, the story would be different. See the San Diego Chargers situation for example. They may end up relocating because of lack of political backing on building a new stadium.