You could say the same thing about tax returns, credit reports, Facebook accounts, medical records, etc. For all these things an employer would benefit by coercing an employee to provide them to the employer. But the risk is worth the reward, so we still have these things. Why is a semi-anonymous voting database any different? I could think of some easy mitigations to your concerns with almost no thought. Make it so that only those who's vote hashes end in a certain number get their private key, so you have plausible deniability. Say 10%. A sample of that size should be enough to verify the vote.

If your employer uses medical records to disqualify you from a position or terminate your employment, get a lawyer because you have a case.

None of those things you mention should be something employers can act on. In many states it would be illegal to factor those things into decisions about hiring, retention, compensation or promotion.