Of course there's a list of common services that you could probably try and they could gain access there like say push/pull on github/gitlab however as long as those common services have another layer of protection (i.e. mandatory commit signing) it should limit the effective attack area pretty effectively.
I also generally find that ssh connections will be one way (i.e. you typically only set up SSH authentication to flow in a specific direction). As long as your SSH authentication graph is directed and acyclic (i.e. no loops and connections only go in one direction), there is little ability for a malicious server to access other nodes in the SSH auth graph provided you connect from a leaf or near leaf node.
I don't use agent forwarding because of the issues with it but there are definitely ways to reduce the attack area that it provides.
This is one of the arguments for generating a unique SSH key on each machine you use. It makes it far harder to break in if you mess up somewhere along the way.