Now when a sending server asks "where should I deliver skywall.example email" by querying MX skywall.example it gets google's servers and starts an smtp conversation with them, saying "I'd like to deliver a message for [email protected]". At this point, Google knows it can accept that, so they say yes, and then continue doing whatever they do to check for spam beyond that, including queries for spf and friends.
The reason the parent suggests this is that if at any point you decide to move off Google, you can pay someone else, e.g. fastmail, for their services, and modify your MX record. 24-48 hours later, DNS around the world catches up and everyone will get fastmail as a response when they ask for your MX record. Any new email goes there instead of Google, and thus you aren't 'tied' to the provider: you just have to move all your old email over. Whereas Google cannot let you move a [email protected] address, because they can only change the MX records for the whole of gmail.
DNS is the source of truth here. Whatever your MX records are is where other servers will try to contact to send email. The MX record is typically just another DNS address that will be queried for AAAA/A (i.e. what is the IP), and that doesn't need to be on the same domain at all.
Here's an example of what it looks like:
delv MX ycombinator.com @9.9.9.9
; unsigned answer
ycombinator.com. 295 IN MX 20 alt2.aspmx.l.google.com.
ycombinator.com. 295 IN MX 10 aspmx.l.google.com.
ycombinator.com. 295 IN MX 20 alt1.aspmx.l.google.com.
ycombinator.com. 295 IN MX 30 aspmx4.googlemail.com.
delv NS ycombinator.com @9.9.9.9
ycombinator.com. 159148 IN NS ns-225.awsdns-28.com.
ycombinator.com. 159148 IN NS ns-1914.awsdns-47.co.uk.
ycombinator.com. 159148 IN NS ns-1411.awsdns-48.org.
ycombinator.com. 159148 IN NS ns-556.awsdns-05.net.
I'll cover off the SMTP outgoing as well while I'm at it. You _can_ also not run your own outgoing smtp server but use someone else's. The key here is that if you use SPF and DKIM, you should put their IPs into SPF and their keys into DKIM, as that is what the receiving server will use. So smtp.zahllos.example could be replaced by sendgrid, provided in my DNS I say so. This may work better, as sendgrid may have a better reputation than the server I chose.
Turns out that I have to treat Apple-included binaries of OSS utilities with suspicion and instead use the version maintained by the Homebrew guys.
1: If anyone else is interested, I wrote a short article about it https://ayewo.com/how-to-get-delv-working-on-macos/
In this case dig or delv would be fine. You can also use https://mxtoolbox.com/SuperTool.aspx and pick "MX lookup" to find the MX records. It will also resolve the IPs, both IPv4 and IPv6 (A and AAAA) for you for the returned records.
Their supertool also breaks down SPF records, for example, into their meaning and tells you if they're valid or not. I think other links were posted that do similar things but I haven't had a chance to try them yet.
I've mentioned reverse pointers in various cases, delv can do those lookups too. Here is how you do it: first you find the IP of say mail-ed1-x529.google.com, which happens to be a mailserver of Google that was the last step before my mail server when sending myself a message from gmail. For a bit of a change, this is an IPv6 only server, and it has IP 2a00:1450:4864:20::529. So let's look it up with -x:
delv -x 2a00:1450:4864:20::529
; unsigned answer
9.2.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.ip6.arpa. 43200 IN PTR mail-ed1-x529.google.com.
One of, but my no means the only, reasons you can't send email from a home connection is because you announce yourself as smtp.zahllos.example but your reverse dns resolves to something like <somegeneratedhostname>.dynamic.residential.isp.com. You can find out yours: type "what is my ip" into google, copy-paste that and do a delv -x on it and see what your "hostname" is according to the internet.
Let's imagine I'm sending from [email protected] to [email protected]. I'm doing it from say Thunderbird or Outlook, and you're using the same.
I need to send, and to do this I typically use an SMTP server. This is something configured, probably on zahllos.example, maybe with the domain smtp.zahllos.example. My mail client contacts this and 'logs in' with my details, then transmits the email message I want to send to this server. The server says 'right fine' and closes the connection.
At this point, the message is in a mail queue ready to go. The SMTP server then does a DNS request for the MX record of skywall.example. Let's say this is 'mail.skywall.example'. My server, smtp.zahllos.example, then connects to `mail.skywall.example', also speaking SMTP, and says "hey, I have this message to deliver".
It is at this point that mail.skywall.example can decide to do some things. It might check SPF, so it will query the SPF record of zahllos.example to find the list of servers that may send email for that domain. In this example, let's assume smtp.zahllos.example is in the list. Great.
It may then also check the mail headers for a signature, called a dkim signature. My server signed the message before sending; mail.skywall.example can query <uid>._domainkey.zahllos.example and find a public key (or not) and check that this signature matches (or not). Again, let's assume it matches.
It might also check something like TXT _dmarc.zahllos.example to see what my DMARC policies are. If I have something like v=DMARC1;p=reject;sp=reject;pct=100;ri=86400;fo=1;aspf=s;adkim=s;rua=mailto:[email protected];ruf=mailto:[email protected];" this tells you I'd like you to outright reject anything not matching policy, that I expect everything to match SPF and have DNS signatures, and you can send reports if you support that to [email protected]. Your server can then enforce these checks as it likes.
One of the first things that will happen is that my server will announce itself via an EHLO statement. An obvious check to do is to check that the sending IP actually matches smtp.zahllos.example. by querying 'reverse ptr' records.
Your receiving server will also likely hand the message over to various spam-checking tools for analysis, such as against DNS blocklists and so on. Larger providers likely have much more sophisticated infrastructure here. Ultimately, you're going to do one of a few things: 1) deliver to inbox or apply user-specified rules and deliver to a folder; 2) deliver to junk (which is typically just another folder, but treated specially by clients), 3) reject, and tell smtp.zahllos.example you don't want the email.
Once the email passes through the smtp dameon, assuming either 1 or 2, it then gets stored somehow and in some way. I'm being a little bit vague here, because 'it depends', but in the simplest scenario, the smtp daemon will write the message to an mbox or maildir-style format. More complex setups definitely exist, indeed, there can be multiple layers of servers doing analysis on separate machines, but for simplicity, mail.skywall.example is one VM that makes its decisions and the result ends up in /var/mail/user@domain/ or some such.
A key aspect of this step that makes email very nice is that if smtp.zahllos.example cannot, for some reason, reach your server now it will queue the message and try again at set intervals. You can reasonably safely turn off mail.skywall.example for a couple of hours.
Another aspect is that you can have multiple MX records where you are prepared to accept email, with priorities. So if you can't accept at one address because the server is down for maintenance, another will accept.
So, now you've technically got an email, but you don't know it. So you open thunderoutlook, and you connect to an IMAP server imap.skywall.example - in our example let us assume this is really the same thing as mail.skywall.example. The server checks you are really you with your credentials. At the backend, this is just another daemon that knows to read /var/mail/... and find new messages; it finds one, downloads its headers and displays it on your screen.
Since we're in a slightly more modern world now, in the case your client was already open you might have an "idle" connection with the server at all times, in which case it can push the message down to you.
In the case of webmail, it is really all the same thing, except you point your browser at a webpage, and that webpage communicates with the servers instead of your client communicating directly. Open source webmail might even use IMAP underneath; things like Zimbra use their own java mail agent, while Google is entirely custom.
That might seem complicated, but in the end it isn't: between two domains at the 'edge', in the end, there's an SMTP conversation. One sending server tells a receiving server it has mail for delivery, and it finds that server by asking DNS where it is. The receiving server may do a bunch of checks against DNS also before making a decision on what to do with the email.