That's a reasonable argument for the people still living in the states. But falls apart when it comes to FATCA.

Also you could make the argument that the people rich enough take advantage of these schemes have already paid more tax than most people through other taxes.

You could also make the argument that if they are rich enough to take advantage of these schemes, they should instead just acknowledge the fact that without the society that gave them the tools to prosper they'd be just some hairless monkeys in the middle of nowhere dying of hunger and that they are thus required to contribute their fair share to the society and community that enabled them to flourish.