Flip comments aside: "ownership" of anything -- a ball, a company, a piece of property -- should be seen as a bundle of rights and responsibilities, and as a (overall) benign socially sanctioned fiction like streetlights. Then you can see that it is only in the trivial cases, like a ball that nobody cares about, that ownership is straightforward and implies "It's mine I can do anything with it I like".
So, duh, you pay taxes and have to conform to zoning codes. Get over it.
EDIT: Also, "government entities" aren't some weird uncontrollable force like the weather or an invading army -- you live in a democracy, so get involved.
If you don't want to support existing infrastructure, I suggest that you become very wealthy, buy an island, and live there by yourself.
The difference between buying and renting is enormously larger than the difference between buying and your theoretical-complete-ownership-of-land. Additionally, in your theoretical ownership world, you would have zero infrastructure to support you and, indeed, would never have had the economic engine to build your wealth in the first place.
Water, electricity, and gas infrastructure can be paid for by taxes on those services, and people who don't want them can not purchase water, electricity, or gas from the "grid". Roads can be paid for with gas taxes, and that way are paid for proportionally by those who use them.
And to add to the GP's post, government can choose to seize your land at any time using eminent domain laws are are more and more abused every day. So, whether or not you agree with Libertarianism, I do think he has a point that there are a lot of similarities between buying and renting that a lot of people maybe don't realize.
I think that the choice of paying property tax and living on an island is a false dichotomy. We could theoretically live in a society without paying property tax, where the government can't seize your land.
One of them is that this "men with guns" phrase is often deployed to plant some kind of "government over-reach" meme around the most trivial of adult transactions, like paying parking fines and property taxes.
It betrays a childish political naivete. The reason I say it's childish is that it's egocentric, as if the concerns of the speaker (to park where they want, to own large plots of land, that abut other people's land, without any consequence or restraint) were the only ones in existence.
That house is served by roads, protected by police and fire services, and must be in compliance with various perfectly reasonable regulations (like brush clearance). That costs money.
And last, you know where your phrase, "All power comes from the barrel of a gun", comes from, right? Good old chairman Mao, one of the top guys in the 20th century rogues gallery. In a democracy, with functioning courts, this saying is more hyperbole.
I think a more relevant notion is that the state is the entity that has a monopoly on violence (due to Max Weber, see http://en.wikipedia.org/wiki/Monopoly_on_violence).
That's a ridiculous simplification. You may as well say, "all power comes from tanks," or, "all power comes from nuclear arms."
If the county takes your house, the sheriff will come out to escort you off the property. Guns don't come into play unless you pose a threat to their lives. Even if guns do come into play, that's still not ultimately the force that moves you. Being outnumbered by a better organized group is what ultimately forces you out. The guns are just one small part of the puzzle.
Now I understand there are some ownership qualities to so-called home ownership, not denying that. There are shades of gray between pure renting and pure owning. But what most people talk about is something clearly in the middle of that spectrum, not on one end. Thus, I think the better description is different variants on renting. Different lease permutations.