This happens, but Last Access Audits usually catch them. Most people don't continue working after they've been termed.

Not a silver bullet, but it covers a majority of these cases.