An election doesn't need to be tamper-proof we just need to be able to detect tampering well enough to make tampering a loser's game.
The problem with randomly hand-counting a few boxes of ballots is that you then need to convince people that the random selection was uniform and fair and actually random.
There are methods to do that, but there are at least as complicated and full of cryptographic finesse, that they ain't simpler than vetting an electronic voting system in the first place.
Having said that: human counting isn't fool proof and is still open to abuse and tampering.
It's mainly that any village idiot can in-theory audit the human-run system, and that it would take a conspiracy with lots of people to engage in wide spread tampering.
The more people involved, the harder it is to prevent leaks.
What they can't trivially do with any system including paper ballots is remove ballots, compared to digital voting machines where you can add e.g. -100 votes to candidtate A, 100 votes to candidate B, thus ensuring that the total-votes field is correct while advantaging candidate B -- this was actually demonstrated by a security researcher on a Diebold touch-screen machine.
With software you either need vetted and approved, very expensive software, or you have to accept a much higher error rate, because the operator cannot verify the results of the process with certainty.