They don't have to use your agent to authenticate again. They just feed commands to your SSH client binary that it then sends over the wire in your existing SSH session-context, with the SSH pty temporarily set to not echo input and discard output (or return it over the network to the attacker, but not render it in the local PTY character-buffer.) A hijacked SSH binary can even detect an elevation-requiring command you provided in the terminal and wrap it in a "sudo -c" along with a payload, such that the whole thing happens as one elevated "transaction", only hitting your agent once.

Happily, this won't work if you only ever use SSH with a non-shell command, e.g. as a fancy encrypted netcat(1).

Right, and that's still a problem, but not as much of a problem as if they were able to then use your key to authenticate to every other machine you have access to. The problem is at least contained to a single system at a time.