It's more than what a Yubikey typically does. There are also can't-be-bypassed audit logs of the key usage, and the manner in which S3 is granted access to the data encryption key is very fine-grained; KMS won't allow the S3 systems to decrypt just anything.

The key is stored in memory while it's being used to encrypt/decrypt, that's unavoidable, but humans don't have access to that. Bucket keys are a bit different, where there's a per-bucket key which has a similar scheme and then per-object keys are derived from it as needed. Together with random nonces/IVs, it ends up being a bit of a mini multi-party scheme.