Yes, you can use ssh-agent with chrome (if they have that implemented, I don't know). Yet you are still in trouble when the chrome sandbox gets compromised, because all your keystrokes are passing through it.
If the sandbox is compromised to have file system access, a process can read your keys from ~/.ssh as well as chrome storage files. Otherwise a webpage has to escape its own sandbox, bypass the native client's sandbox (in a different process) composed of the inner and outer sandbox and then access the native client.
I'm not saying that it's impossible, I'm saying that using a simple analogy as "a house with two doors" might not be the best.
Or there could be just some really dumb bug that somehow enables cross-process access. With javascript. You know, one of these silly brown-paper-bag bugs that are not supposed to happen.
Either way, this is the second door. It may be a shiny steel door, but it's an additional door.