You should account for deletion, but in general this stuff doesn't keep me up at night.

In the rare event you restore deleted users, and even rarer, if they actually notice, just tell them you screwed up.