>I’m not entirely convinced it’s fair for non residents to vote, especially those that have no intentions of coming back

I think it would be fair to be able to opt out of tax filing obligations in the US and for that give up the right to vote until you return.