Extradition policies, consumer protection laws, intellectual property status (ie whether you can use a tax haven incorporation to shield yourself from patent lawsuits or even to use it to hoard patents so you can use it as a base for launching law suits on others, forcing them to come to court in the tax haven jurisdiction for disputes)
There are so many things tax havens are used for. And that is why incorporating inside of the country where they are trying to get bailouts from is so important, otherwise the tax payer's money will be used against their own interests, with no way of enforcement.
Tax haven registered companies are the Nigerian Prince's of the corporate world. It is very possible they might be legit (and some very legit businesses are tax haven registered), but the very fact that they registered there on purpose is very telling in itself. Telling you they can swindle you with or without your consent.
"Does not break any rules"
and
"Follows all the rules"
Where the gap is in whether the entity is subject to the rules. You are correct that one that isn't subject to the rules is right to not follow them.
But, where the outcome of those rules incurs some benefit, why should the non-subject incur it?
Minimizing tax liability is not necessarily doing anything outside the law.