1) Verify that my vote was correctly encoded and cast as intended. Every voter must verify this. It can be as "easy" as: printing a hash produced by a voting terminal/device you trust (which doesn't need to be government provided), submitting the resulting encoded vote into a (untrusted) government website where you authenticate with a smartcard, and then later comparing your original hash against a hash published in a newspaper... This exceeds the level of sophistication/care of most voters in any large country.
2) Some organization must check a large trail of signatures, zero-knowledge proofs of correctness, re-encryption mixnets, etc, etc. This can be done well enough as long as every political party and election watchdog has one or two crypto professionals in their employ.
I am not really worried about #2, even though it is harder, I am worried about #1. The important part is that all the checking of the world regarding #2 is useless without at least a representative subset of voters performing #1 (~1% would be enough, but only if distributed uniformly at random, otherwise you can flip votes from the population(s) least likely to check).
For an analogy: I grew up in Mexico during the 70 years PRI rule, and I can tell you in my city, in my polling station, with watchers from all major parties and a few ONGs, it is actually very unlikely that anyone stuffed the ballot. But in a rural area, where Spanish is not the primary language and the poll watchers were all from one party or few enough to be for sale... well... My point being that a similar scheme is quite possible when the barrier is "computer literacy" instead of "rights literacy" or plain old literacy (in the country's majority language, I mean). Either way, the smaller the group of people who understand how the election works, the easier it is to disenfranchise people.
There's little problem in an election where only people with knowledge of cryptography can verify the official results.