I have a 6 digit pin (8 digit admin pin) with 3 retries before the device wipes itself. The chances of an attacker cracking that are negligible - I'm more concerned about someone observing my typing (applies to passphrase as well) or actual physical attacks on the yubikey (stolen, decapped, ...) but I'm not the target demographic where such effort would be warranted (or at least I hope so). An attacker could backdoor my PC and intercept the passphrase dialog - but in that case he'd still need my yubikey which makes remote attacks impractical and makes me notice the theft. This is not the case with a passphrase on a secret key - a machine compromise may silently take everything without you ever noticing.