Of course you do, it just takes a teeny bit more work--you swap the contents of crypto.js with the contents of malicious-crypto.js. The threat model is exactly the same (users either trust the server or they don't) but the browser crypto option adds a layer of (respectfully, because I know you're a very intelligent person who means well) bullshit.
I think the question of where your password by design will go is very important. If it's transiting the server, suddenly there's a lot more surface area to worry about. Logfiles, databases, and the like suddenly can be called into scope, and an attacker might be able to steal credentials even without being able to substitute out code.
In any case, the great thing about an open ecosystem is that, if you don't like the choices someone else has made, you are more than welcome to make your own implementation with choices you prefer!
As soon as you're in-browser, you have a choice: do it server-side, which means you have access to people's funds (at least while they're logged in), or do it via client-side JS, in which case you don't. The choice there for Stellar ended up being pretty straightforward, though the simple implementation leads to some UX surprises such as: https://www.stellar.org/faq/#_Why_do_I_need_to_authenticate_....