For clarity: no one signs a ballot in the US. States with remote voting options generally have a ballot envelope which must be signed and sealed. That is: you have to authenticate that the ballot was cast by the right person, but the vote itself can only be tallied anonymously.

This is sort of a dumb idea, really. But it's not as broken as you imagine.