What sorts of problems do you run into? I tried to set up this exact thing (PIV mode) a few days ago, and apart from the fact that gpg-agent won't accept the SSH keys for the card (which can be worked around with a simple alias), things have been great.