You only need to sign the packets in the key exchange on the card. The normal payload packets are protected by symmetric algorithms based on the ephemeral symmetric key generated in the key exchange, no need (and no use) to involve the smartcard there.