But for this class of algorithms, that doesn't have to be the case. Starting with a cryptographic hash could keep the key from being reversed to the input. There are probably other things that could help as well, like maybe not using the password in the public key at all.
Remember, the attack is: start with "aardvark", run the deterministic key-generation process, hash, scrypt, elliptic curve and all, and check to see if you get the same result. If you don't, move on to "apple", then "ataxia" and so on.
The attacker has every piece of information in the system except the password string, which they can substitute with a dictionary.
In most systems where the server never sees your password, they store a hash that is generated from the password.
Then the login code on the client takes the password, performs the same operation and then sends either that same hash to the server, or one that's simple to calculate from it.
So if I steal your password file, I can log in as everybody without even knowing what their passwords are. I just change the code in the client to skip the hash generation part of the authentication algorithm and use the one I supply.
The idea with asymmetric ciphers (at least the ones that interest us) is that one party HAS to know the secret, and the other has no way to get it.
I think you're saying that this system is better because, since it uses public key pairs, an attacker who steals the password file doesn't know enough to log in as a different user.
Leaving aside that an attacker who steals your password file has probably mooted login security altogether, the attacker in this system seems to have exactly the same leverage as an attacker who steals a password file full of scrypt hashes.
Instead of sending the public key every time, the server sends a challenge, which only the private key holder can answer. This is a pretty basic usage of keypairs and an extremely brief perusal of their source code suggests to me they've thought of it.
So no, the public key is not the password, unlike hash based systems which do suffer from the problem you're concerned about. The hash IS in fact now your password in those systems.