Why couldn't a non-resident security key send it's public key as username? And the response contains the actual username and private key.

Privacy. The idea, IIRC, was to have separate identifying material for each site.

Because the security key doesn't store any public keys.

Basically, the security key stores a single symmetric key. It'll generate a public/private keypair on registration, encrypt it, and send it to the server. On authentication the server will return the keypair back to the security key, which decrypts it and uses the retrieved private key for authentication.