You still need to allocate physical space for 1) the cryptographic circuitry of the token 2) the tamperproof and stress-resistant enclosure. Depending on the form factor, you might not have that much physical space left for storage. The physical requirements for a token are much more complex than a plain USB drive.

Google’s security key can already support 250 resident keys. I’m not sure how secure it is compared to Yubikeys, but it can’t be that far behind.