It most definitely defeats the purpose. Whether it's worth doing depends on how much you value your time and how secure you think the rest of your setup is (ie. if it's super secure the marginal security might not be worth the additional time).
Why?
If your password manager password is compromised, you're pretty screwed no matter how you slice it. For most people that use a password manager, I would guess that exposing their main password is an unlikely scenario. Loss/theft of a phone seems much more likely, and in that scenario, you're exactly as screwed as you would be if you had all of your 2fa codes in your password manager.
I think it's pretty valuable - you automatically get the benefit of your pw manager recognizing the domain and only logging you in/copying the code if the domain matches what's in the password manager.
It's already:
Type in password expecting a smooth login > Process that the new empty text box means you have to check your MFA source > Realise you have to find your phone > Find the authenticator app > Look through the (mine is long) list of registered accounts > If there are duplicates (e.g. AWS Account X, AWS Account Y) then select the right one > See if the timer is nearly 0 and if you'd like to risk using the code displayed > Enter code