Himitsu's ssh-agent does the same, and if you use some agent that uses a secret-service backend, you'd get the same result.

It's mostly the default ssh-agent that has a different security model.