SSH Agent Forwarding does not happen by default. You need to include the -A option in your ssh command, unless maybe you've enabled it globally in your ~/.ssh/config file.

They can't get your private keys, but they could "perform operations on the keys that enable them to authenticate using the identities loaded into the agent" (quoting the man page). This would also only be possible while you are connected.