Rich enough to have money that can be taken away, not rich enough to afford a specialist for tax evasion. People like this pay most of the tax.

And yes, all the reasons why "taxing X would actually be harmful for the economy" somehow magically do not apply to the small fish.