Your public key, however, would have one purpose and one purpose only - to identify you. As the public half of an asymmetric crypto pair, and given that people share their public keys on keyservers and on webpages all the time, it wouldn't be too difficult to convince an organisation that wasn't aware of the issues to give you the public keys associated with their accounts.
With that information, it would be really easy to definitively tie your identities together because you only need the public key in order to do so. Very few other pieces of information taken in solitude can do that - names are not unique, passwords are not unique, even physical addresses are not unique. (I'll grant that email addresses might be, but even then, companies aren't going to hand out email addresses to anyone who asks because of spam.)
That's why I'd use a different key per site.
I always wondered, why password-bases authentication is so prevalent, when asymmetric cryptography is there and actually used under the hood. That's a dream: one key to rule them all and no security problems with leaked accounts, unified UI to register, login, logout.