The handing of the keycard is necessary to ensure it's destroyed and can't be used as a "proof" you work somewhere (most access cards these days have your name, face and the company logo printed on the front).
The keycard will be removed from the access list to the building even when it's destroyed, they're not considered reusable by most companies.
Your private key is not reusable, it should be destroyed and revoked from all system when you leave a company.
Aside from a third party attacker (which is well-covered by my normal practices), that's a threat model that I'm personally not worried about at all, really. In part because I've never seen or heard of that happening and in part because if it did, I am confident that there are enough records to be able to prove it.
There are exceptions but the circumstances where an employer would need to retrieve my keys without my assistance are extremely rare and in those instances it’s unlikely I’d still be an employee anyway.