Yeah, that's me. So I have started experimenting with the Yubikey 4. (For those who don't already know: the Y4 keeps the private key somewhere in hardware, and never lets it out; the Y4 itself does the TOTP calculation onboard). The Windows desktop Yubico Authenticator (which talks to the Y4) is not too painful.
Both allow effectively unfettered access to your Google account (with the exception of some account security changes, for which both require your password), both can be revoked from the web interface (arguably a TOTP secret is worse as it can create new session tokens but revoking it doesn't revoke tokens it created). Both can be stolen from unencrypted flash, and stealing either requires root (or privilege escalation, which is effectively root).
This whole "TOTP is weak, use yubikey" argument seems 100% about convenience, with a smattering of pseudo-security backing it up.
Why would you trust Google or anyone else with your work (bank, Github, whatever you use TOTP for) credentials? I can't understand how anyone would think it's "pseudo-security". As you rightly say, your credentials are in plain text on that phone.
That's a good description of the state of my knowledge.
To answer your question: the Google account that my phone has access to doesn't have anything important, so I'm not worried about any related keys leaking. The TOTP secret would be for something more important, that the phone itself doesn't have access to.
In other words, I'm wondering whether a generic Android phone is safe to use as an over-sized hardware security token (or rather, as backup for a more convenient hardware token). It seems the answer is no.
For me, I trust my phone more than any other piece of computing equipment I own. I also rarely unlock my password manager on it, so there's no one attack that gets you everything you need by hitting either my phone or my computer.
If I had a generic, non-Google Android phone, or any kind of rooted phone, the equation would be different for me.