This is what I do too, I've written about it in detail at: https://nickjanetakis.com/blog/using-a-custom-ssh-key-to-acc...
You can also set that in your `.git/config` file on a per repo basis. Additionally you can set `GIT_SSH_COMMAND="ssh -i /tmp/custom_key_ed25519 -o IdentitiesOnly=yes" git pull` to override whatever is in your global or repo config for 1 off pulls as a different user.
put this in your ~/.gitconfig-personal
and this in your ~/.gitconfig-work