Remove it from both. However make sure that you have quite a lot of backups of your 2FA backup keys, and maybe even one offline backup of your seed, if you lose them, the account is gone (which is a good thing, I guess).

Thanks. I have Google backup codes as well as multiple Authy installations, Google prompt and a recovery email address so I guess I should be covered :)

What is "Google prompt" here?

It’s a notification pop-up that the Google mobile app can send, asking for login confirmation.

I see, but doesn't have the same issue as sending an SMS to the phone if the phone has been stolen in that the thief can just say "yes" to the prompt?

No, it's not quite the same issue. SIM swap works remotely without anyone getting a hold of my phone.

A stolen phone can of course be a problem as well, but at least it's somewhat under my control and I may notice pretty quickly when it's gone.