Yep, this is an endemic problem with end-to-end verifiable elections everywhere: it is easy to invalidate whole segments of votes. For example, in the wombat voting scheme, if one of the people who holds a decryption key refuses to divulge it, perhaps by "losing" their key, they can invalidate the whole election. Typically to minimize the chance of this happening you make sure that everyone with a key has a strong commitment to the election and make not divulging the key a public shaming event. "OMG, the Democratic Party refuses to decrypt the vote in this Republican-dominated area -- they must be corrupt!"

It's an awfully sticky situation, because the A (availability) in the CIA principle is so critical in elections, where it typically is compromised the most in encryption schemes. (i.e. Quantum Key Distribution ensures that Eve can't listen in on communication, but doesn't do much to stop Eve from making communication impossible).