This is a little confusing since it’s not just an access token (or refresh token) since the OP wants to verify that the sender is the correct sender, not just that they have access to the token. Otherwise a token holder could simply pass the token to spammers, still. This is harder since the sender could share his or her sender key, so you’d have to disincentive that, but doing so it going to make the system even more complex.

You don't have to go very far with extra checks and disincentives, as the recipient can also invalidate the token at the first sign of spam. SPF/DKIM or an equivalent can also help with sender authentication.

I think the idea is that OAuth token would allow sending emails as a specific sender, like if I allow say Lyft to send me emails in this flow and the pass the key to spammers the emails will appear to come with Lyft.

I actually loved XMPP/Jabber flow where sender first need your permission to get added to the contact list and get their messages thru. Still can be spammed but without any meaningful content at least.