That is a true point :) But keep in mind that there is something to be said for not storing encrypted data on the same machine as the key.

Ofcourse once the machine is compromised it can trick you into giving it the key, but there are also countless scenarios where that is not possible. It is really an improvement onto your security.

Lets not pretend that read-protected memory is such a secure way of storing private keys.

If JS sent by server has access to your key, then by that proxy the server has access to the key too. The security is equivalent to having key stored in encrypted form and decrypted only for the duration of the session/use and the password not being stored on the server.