Yes, the mailman could commit a federal crime and figure out how you voted for. So if you're worried about that, drop the envelope in the actual ballot box.
The mailman sees even more. If you live in a condo and drive a Prius with a "Coexist" sticker, you probably voted one way. If you live in a single-family home with a big yard and you drive a pickup with a "Blue lives matter" license plate holder, you probably voted the other way.
Your party affiliation is already public record, no need to attempt to infer your preference from your name or address.
For reference, here is how it works in Washington state: https://www.sos.wa.gov/elections/faq_vote_by_mail.aspx
Then the ballot is placed in the same hoppers that the in person votes are put in and they are commingled.
To do the correlation in person you would have to do it in the polling place, on election day, in front of election monitors and the voter, during the 30 seconds between when the person fills out their ballot and when they drop it into the box.
If your name is on the envelope, now it can be done anywhere between your mailbox and the polling place during the period of days or weeks between when you put it in your mailbox and when it gets counted. Wouldn't we then need independent election monitors guarding the incoming mailboxes 24/7 for several weeks?
The outermost envelope is verified and opened, and the innermost envelope is collected with all of the others in a different area for counting.
My town has a bunch of dropboxes which are picked up by election volunteers if you still don't trust the mail. An election volunteer could "lose" a box of ballots but that seems like it would be pretty easily detectable as well.
Since mail tampering is a federal crime maybe Trump could pledge to pardon every mail carrier who throws away ballots from likely Biden voters. Seems pretty unlikely, but I don't know how to counter that.