I don't need a Math PhD to use all of those. But if I want to be audit it? Probably. I can audit paper ballots just fine, anyone can. But that easy to use shell? How do I know that it does what people claim it does? I have to trust those people. The people that have the ability to audit now hold absolute power over the result

You can make an interface that hides the complexity and lets people check their votes in a user-friendly way. There's nothing that makes it fatally complicated to use.

As I said: The problem is not using the software itself but knowing that the green checkmark (or whatever) it displays actually means the vote was correctly counted. The former is trivially done by a competent team, the latter not so much.

Just an example from another field: It is easy to compare key fingerprints shown by WhatsApp/Signal/GPG but how do I know that the software does not encrypt the text with a second key? Personally I am not competent to do so and have to trust others. Don't forget that it's not enough to audit the code itself but you would also need the check the whole build process from scratch (and processors it was done on for that matter - and while we are at it: how can I know that the display actually shows that it was asked to do?). How many people know how to do so?

Hiding the complexity is exactly the wrong approach. You want to reduce the complexity so that the average person can understand it just fine.

How do you verify that the machine actually does what it displays and/or print without hooking a hardware debugger on it?