As a counter-example of having to trust the receiving end, consider zero knowledge proofs [1]. Even if you don't trust the prover, you become convinced of what they're trying to prove to you (if it's true).
The basic trick to proper cryptographic voting machines is zero-knowledge-proving that they are working correctly. Any tampering, whether it be in the source code or at the hardware level, will either not affect the tally or break the zero knowledge proofs.
Everybody understands paper voting and there are several (more or less) trustworthy protocols for paper-based vote count.
Maybe I'm pessimistic, but I can't picture a near future where the populace understands computers enough to be able to trust e-voting.
All the 'did I vote for the person I tried to vote for?' happens either in the voting booth, or by choosing unused ballots to audit at random.
If vote is secret you just know that someone voted X. Who is that someone? Is he even real?
Since we need secrecy we need to know X, but not who is "someone", so we're blind there.
If you can't prevent fake voters from registering, you've failed before the election even begins.
(Yes, this does allow you to coerce people with known preferences into not voting in order to affect the result. This issue, and many others, are discussed in the video.)