It most definitely defeats the purpose. Whether it's worth doing depends on how much you value your time and how secure you think the rest of your setup is (ie. if it's super secure the marginal security might not be worth the additional time).
Why?
If your password manager password is compromised, you're pretty screwed no matter how you slice it. For most people that use a password manager, I would guess that exposing their main password is an unlikely scenario. Loss/theft of a phone seems much more likely, and in that scenario, you're exactly as screwed as you would be if you had all of your 2fa codes in your password manager.
I think it's pretty valuable - you automatically get the benefit of your pw manager recognizing the domain and only logging you in/copying the code if the domain matches what's in the password manager.