It's not a vanilla stretched key that gets send to the server. It is, in effect, the equivalent of your public key (that just happens to be kept secret by the server so it can use it to prove its identity, as the bearer, to you)

I think the use of that is really just isolating the cross site hacking at that point. You are absolutely correct that if they have your password, its game over on that particular site.