The problem with token reuse is the same as with password reuse: If a site gets compromised, your token is worthless. If the token is burned into hardware, then your hardware is now worthless.
I didn't mean that tokens should be shared across sites; more that a single physical token for a role account (like a backup admin login for an auditor could be escrowed with a CFO (who does not have a login)
You'd still have one hard token per site (in reality, you'd have one or two hard tokens for the most important things, and then use soft tokens for everything else.)
(If I'm logging in primarily from a phone/tablet, an authenticator app on the same device is much less secure against targeted attacks than a hardware token would be. Plus, hardware tokens allow lots of useful things like physical-escrow based access control.)