WireGuard seems to use symmetric key crypto for packet encdec. The card would need to sign only the handshake, which occurs "every few minutes"[1] and "is done based on time, and not based on the contents of prior packets"[1].

1: https://www.wireguard.com/protocol/

You only need to sign the packets in the key exchange on the card. The normal payload packets are protected by symmetric algorithms based on the ephemeral symmetric key generated in the key exchange, no need (and no use) to involve the smartcard there.