There is a cryptographic primitive that does just this allowing of plausible deniability whilst ensuring that only persons able to vote voted: Using Linked Ring Signatures.