We could also use open source hardware that isn't pen and paper. We just need the open source part.

What open source hardware would that be, and how can everyone eligible to vote be able to understand how it works and be able to observe that it really does work?

Every single person doesn't understand all the security aspects of today's paper & pen ballots, how they are secured before and after the day of voting, how they are transported from one location to another, how mail-in ballots get handled, etc.

There's no reason to require something that everyone can understand when talking about a different voting system.

IMHO, we should aim to build something (whether using physical machines or using cryptography/mathematical properties) that the various parties in a government can audit and recognize as secure and that leads to a larger portion of the population voting or a cheaper election process.

Whether people do understand something is very different from whether people can understand something.

While many people maybe aren't aware of the security mechanisms of a pen and paper election, essentially everyone can actually understand them if they feel the need to, and that is extremely important. You cannot resolve widespread distrust by putting a bunch of mathematicians on TV who tell the population that everything is fine.

We already have something that can be made secure, anonymous, and auditable, and yet is easy to understand and relatively inexpensive. It is pencil and paper ballots, with multiple interested parties observing the entire process.

And a way to simply audit them. It's a bit difficult to make a fake pen and paper that would trick me into voting for the wrong person (especially at scale).

Well then you have issues like "hanging chads," [0] that's why e-voting became big in the US.

0: https://cseweb.ucsd.edu/~goguen/courses/275f00/abc-chads.htm...

Pen and paper doesn't suffer from hanging chads, surely? That's only if you use paper + hole punch for primitive automated counting?

Nope, that would not work. Paper and pen are the best secure methods. Everything digital is insecure.

Verifying hardware more complicated than a pen and paper is prohibitively difficult.

remember the butterfly ballot?

never forget the hanging chad!