i attempt to use this and some programs recognize this and many just don't

Don't these apps just use PAM? Since the initial complaint was about sudo, I'd figure pam / polkit would handle this, and apps would call those to obtain privilege elevation.

Forwarding the ssh agent (-A) is considered insecure. Instead man ssh recommends using a jump host (-J)

I was talking about the GPG agent, so that the key on the smart card can be used to for sudo elevation on the remote host. This usually requires user interaction with the key, so just having access to the agent wouldn't do much. I don't think the ssh agent would help with this.

To your point, I wonder whether that consideration holds when the private key is held on an external device, like is the case with a YubiKey. I use that setup, and I can't add the key to the ssh agent.

    $ ssh-add .ssh/id_yubikey_gpg.pub                                                                                                                                                                                                                                                                                                                     
    Error loading key ".ssh/id_yubikey_gpg.pub": error in libcrypto

SSH connections work fine with that key.