Your reasoning is ignoring that billionaires disproportionately benefit from public investments (tax money). Therefore, when they avoid paying taxes, they're taking more from the community than they are putting back in: this is theft, and degrades society for everyone (even billionaires, in the long run).

Better neighbor analogy: your neighbor asks for small favors all the time, and you provide. One day you ask him for one and he leaves the neighborhood entirely instead of obliging.