> Voting via your phone would save huge amounts of time and energy compared to paper voting and it would make holding these votes much easier.

This would open up a whole can of worms for things like your employer (or other entity) being able to verify you voted "correctly", excluding people without smartphones, require ID to vote, &etc...