In a properly designed end-to-end verifiable voting system, you don't need to verify the code. Black boxes work just fine, you need to be able to verify that the output had indeed matched your input.

I'm not advocating use of computers, though. Believe paper ballots with some math attached are the way to go. That way there are both anti-fraud properties baked in the system (one can verify that their vote wasn't messed with), and the classic hard copies so any person who can do some basic arithmetic can count and re-count votes just fine.

Unfortunately, I think all the systems I've read about either were found to have some issues (usually, it's about vote secrecy) or just too new (essentially, not so well reviewed).