News for hackers. Stuff that matters.
Re:After self-hosting my email for twenty-three years I have... by skywal_l on Sunday September 04, 2022 @03:50PM (#32717018)
I am ashamed to admit that I have no idea how email works. Is there a dumb down explanation of what are the moving parts and how you can achieve that sweet spot?
Re:After self-hosting my email for twenty-three years I have... by nvahalik on Sunday September 04, 2022 @03:57PM (#32717085)
You send your email through a client. That client then sends (transfers / SMTPs) that email through an MTA either bundled with it or provided by your mail server.

The MTA parses the message, figures out who it needs to go to (To, CC, BCC headers), figures out what servers receives mail for those recipients, and then transfers (SMTPs) it to the server.

What OP is referring to is that the MTA essentially does a DNS lookup for the recipients domain for a record of type MX (Mail eXchanger).

If you own the domain you have complete control over where that mail goes: you own the MX record.

Re:After self-hosting my email for twenty-three years I have... by jeffbee on Sunday September 04, 2022 @04:05PM (#32717159)
There is no such thing as a BCC header. That's the entire point of the BCC.

What you've described might be correct in some cases but is not universal. The mainstream way that messages are sent is your "email client" or MUA speaks the ESTMP protocol to a mail transfer or submission agent (MTA or MSA). The client directly specifies the envelope recipients, which are not, generally speaking, parsed out of the formatted message. That is why it is possible for me to send a message to myself but the message is subsequently delivered to hundreds of unnamed recipients.

Re:After self-hosting my email for twenty-three years I have... by megous on Sunday September 04, 2022 @05:45PM (#32717998)
> The MTA parses the message...

That may happen, but not for the purpose you stated. MTA just uses addresses you told it to use in the SMTP dialog. It doesn't use addresses from the message. In fact the addresses in the message may be totally different.

Re:After self-hosting my email for twenty-three years I have... by patmorgan23 on Sunday September 04, 2022 @06:08PM (#32718188)
Email is complicated but the jist is there are relays and mailbox/mail-exchanger servers (and email clients). And a million different anti-spam measures that make making sure people actually get you email difficult.

When you send an email you mail client contacts you mail-exchanger (MX) and drops the mail in your outbox. Then the MX will look up the MX record for the domains in the TO field and attempt to send the email to it using SMTP.

The first thing receiving server ussally does and look up the IP of the sending server and see if it's on a spam black list. If it is it will probably just drop the connection. Then it will look up the SPF record for the domain in the FROM address and see if the sending server is allowed to send that mail.

Larger email services will have an internal 'reputation' scoring system that will use data from reported spam to figure out what IP's and domains are sending spam emails and filter them out. They'll also look at if it's a residential IP, in an IP pool for a major cloud provider etc. Each provider has a different system and it can be really difficult to get a provider to trust your IP or whitelist your IP so you can make sure that your mail actually gets to who you're trying to send it to.

Relays are pretty simple they'll take email from one place and send it to the recipient. The mail exchanger usually has a built-in relay. A lot of people will use a third party relay service so they don't have to worry about managing the reputation of the IP of their sending mail server. They'll just add an SPF record for the relays service to their domain. And then configure their mail exchanger to send all outbound mail to the relay and then the relay will do the MX lookup.

A lot of mail services will also have their MX records pointed at relays. These inbound relays will often have a lot of those anti-spam services bolted onto them and they can also be used for load balancing to make sure that the service is always able to accept mail even if it doesn't make it in the mailbox immediately.

Re:After self-hosting my email for twenty-three years I have... by delecti on Sunday September 04, 2022 @04:01PM (#32717119)
Typically, if you sign up for an email account, you get an email address like [email protected] or [email protected]. Alternatively, if you own/host skywal.com, you can have an email address like [email protected] served from a computer in your home.

The "sweet spot" is combining the two, where you own skywal.com, and have your email send/receive through Google or whoever. Then, if Google decides to ban you, you just register skywal.com with another company who provides that same service, and you keep your same email address.

That's the broad strokes anyway.

Re:After self-hosting my email for twenty-three years I have... by zahllos on Sunday September 04, 2022 @05:13PM (#32717785)
I can type out an explanation relatively easily.

Let's imagine I'm sending from [email protected] to [email protected]. I'm doing it from say Thunderbird or Outlook, and you're using the same.

I need to send, and to do this I typically use an SMTP server. This is something configured, probably on zahllos.example, maybe with the domain smtp.zahllos.example. My mail client contacts this and 'logs in' with my details, then transmits the email message I want to send to this server. The server says 'right fine' and closes the connection.

At this point, the message is in a mail queue ready to go. The SMTP server then does a DNS request for the MX record of skywall.example. Let's say this is 'mail.skywall.example'. My server, smtp.zahllos.example, then connects to `mail.skywall.example', also speaking SMTP, and says "hey, I have this message to deliver".

It is at this point that mail.skywall.example can decide to do some things. It might check SPF, so it will query the SPF record of zahllos.example to find the list of servers that may send email for that domain. In this example, let's assume smtp.zahllos.example is in the list. Great.

It may then also check the mail headers for a signature, called a dkim signature. My server signed the message before sending; mail.skywall.example can query <uid>._domainkey.zahllos.example and find a public key (or not) and check that this signature matches (or not). Again, let's assume it matches.

It might also check something like TXT _dmarc.zahllos.example to see what my DMARC policies are. If I have something like v=DMARC1;p=reject;sp=reject;pct=100;ri=86400;fo=1;aspf=s;adkim=s;rua=mailto:[email protected];ruf=mailto:[email protected];" this tells you I'd like you to outright reject anything not matching policy, that I expect everything to match SPF and have DNS signatures, and you can send reports if you support that to [email protected]. Your server can then enforce these checks as it likes.

One of the first things that will happen is that my server will announce itself via an EHLO statement. An obvious check to do is to check that the sending IP actually matches smtp.zahllos.example. by querying 'reverse ptr' records.

Your receiving server will also likely hand the message over to various spam-checking tools for analysis, such as against DNS blocklists and so on. Larger providers likely have much more sophisticated infrastructure here. Ultimately, you're going to do one of a few things: 1) deliver to inbox or apply user-specified rules and deliver to a folder; 2) deliver to junk (which is typically just another folder, but treated specially by clients), 3) reject, and tell smtp.zahllos.example you don't want the email.

Once the email passes through the smtp dameon, assuming either 1 or 2, it then gets stored somehow and in some way. I'm being a little bit vague here, because 'it depends', but in the simplest scenario, the smtp daemon will write the message to an mbox or maildir-style format. More complex setups definitely exist, indeed, there can be multiple layers of servers doing analysis on separate machines, but for simplicity, mail.skywall.example is one VM that makes its decisions and the result ends up in /var/mail/user@domain/ or some such.

A key aspect of this step that makes email very nice is that if smtp.zahllos.example cannot, for some reason, reach your server now it will queue the message and try again at set intervals. You can reasonably safely turn off mail.skywall.example for a couple of hours.

Another aspect is that you can have multiple MX records where you are prepared to accept email, with priorities. So if you can't accept at one address because the server is down for maintenance, another will accept.

So, now you've technically got an email, but you don't know it. So you open thunderoutlook, and you connect to an IMAP server imap.skywall.example - in our example let us assume this is really the same thing as mail.skywall.example. The server checks you are really you with your credentials. At the backend, this is just another daemon that knows to read /var/mail/... and find new messages; it finds one, downloads its headers and displays it on your screen.

Since we're in a slightly more modern world now, in the case your client was already open you might have an "idle" connection with the server at all times, in which case it can push the message down to you.

In the case of webmail, it is really all the same thing, except you point your browser at a webpage, and that webpage communicates with the servers instead of your client communicating directly. Open source webmail might even use IMAP underneath; things like Zimbra use their own java mail agent, while Google is entirely custom.

That might seem complicated, but in the end it isn't: between two domains at the 'edge', in the end, there's an SMTP conversation. One sending server tells a receiving server it has mail for delivery, and it finds that server by asking DNS where it is. The receiving server may do a bunch of checks against DNS also before making a decision on what to do with the email.

Re:After self-hosting my email for twenty-three years I have... by skywal_l on Sunday September 04, 2022 @05:21PM (#32717833)
So in the example of the parent, he's got a domain name registered somewhere (mydomain.com) and he set, in its MX records, the gmail server. But how does gmail make the connection between that address and your gmail address then?
Re:After self-hosting my email for twenty-three years I have... by zahllos on Sunday September 04, 2022 @05:39PM (#32717945)
So normally in the cases like this, you also have to tell Google about this, and you typically do this by using one of their paid-for products like workspaces or apps for business or whatever they call it. So let's say that you decide to host [email protected] with Google. You pay them for workspaces and you likely tell them "I would like to use this domain I already have, with email". They then tell you "OK, add our servers as your MX record in your DNS, or transfer the whole domain to us and we'll do it for you". In this case we're doing the 'changing the MX records' part.

Now when a sending server asks "where should I deliver skywall.example email" by querying MX skywall.example it gets google's servers and starts an smtp conversation with them, saying "I'd like to deliver a message for [email protected]". At this point, Google knows it can accept that, so they say yes, and then continue doing whatever they do to check for spam beyond that, including queries for spf and friends.

The reason the parent suggests this is that if at any point you decide to move off Google, you can pay someone else, e.g. fastmail, for their services, and modify your MX record. 24-48 hours later, DNS around the world catches up and everyone will get fastmail as a response when they ask for your MX record. Any new email goes there instead of Google, and thus you aren't 'tied' to the provider: you just have to move all your old email over. Whereas Google cannot let you move a [email protected] address, because they can only change the MX records for the whole of gmail.

DNS is the source of truth here. Whatever your MX records are is where other servers will try to contact to send email. The MX record is typically just another DNS address that will be queried for AAAA/A (i.e. what is the IP), and that doesn't need to be on the same domain at all.

Here's an example of what it looks like:

    delv MX ycombinator.com @9.9.9.9
    ; unsigned answer
    ycombinator.com.        295     IN      MX      20 alt2.aspmx.l.google.com.
    ycombinator.com.        295     IN      MX      10 aspmx.l.google.com.
    ycombinator.com.        295     IN      MX      20 alt1.aspmx.l.google.com.
    ycombinator.com.        295     IN      MX      30 aspmx4.googlemail.com.
This is me using DELV to ask "where should I send email for ycombinator.com?" and I have four responses. Column 5 tells me the priority. Lower numbers are higher priority. Unsurprisingly, this is Google. But let's see where they host their DNS, shall we?

    delv NS ycombinator.com @9.9.9.9
    ycombinator.com.        159148  IN      NS      ns-225.awsdns-28.com.
    ycombinator.com.        159148  IN      NS      ns-1914.awsdns-47.co.uk.
    ycombinator.com.        159148  IN      NS      ns-1411.awsdns-48.org.
    ycombinator.com.        159148  IN      NS      ns-556.awsdns-05.net.
So AWS. So they have separate DNS to Email, and could change those MX records to host their email anywhere else, without needing to change or move ycombinator.com's DNS from AWS.

I'll cover off the SMTP outgoing as well while I'm at it. You _can_ also not run your own outgoing smtp server but use someone else's. The key here is that if you use SPF and DKIM, you should put their IPs into SPF and their keys into DKIM, as that is what the receiving server will use. So smtp.zahllos.example could be replaced by sendgrid, provided in my DNS I say so. This may work better, as sendgrid may have a better reputation than the server I chose.

Re:After self-hosting my email for twenty-three years I have... by ayewo on Monday September 05, 2022 @12:47PM (#32726338)
I learnt about delv for the first time reading your comment and struggled [1] to get the commands you used here to work on my Mac.

Turns out that I have to treat Apple-included binaries of OSS utilities with suspicion and instead use the version maintained by the Homebrew guys.

1: If anyone else is interested, I wrote a short article about it https://ayewo.com/how-to-get-delv-working-on-macos/

Re:After self-hosting my email for twenty-three years I have... by zahllos on Wednesday September 07, 2022 @04:10PM (#32756078)
delv is the successor to dig, and both are tools that come as part of BIND the DNS server for making queries and debugging DNS. Delv happens to understand DNSSEC a bit better, so if you ever need to debug that, it is handy to have.

In this case dig or delv would be fine. You can also use https://mxtoolbox.com/SuperTool.aspx and pick "MX lookup" to find the MX records. It will also resolve the IPs, both IPv4 and IPv6 (A and AAAA) for you for the returned records.

Their supertool also breaks down SPF records, for example, into their meaning and tells you if they're valid or not. I think other links were posted that do similar things but I haven't had a chance to try them yet.

I've mentioned reverse pointers in various cases, delv can do those lookups too. Here is how you do it: first you find the IP of say mail-ed1-x529.google.com, which happens to be a mailserver of Google that was the last step before my mail server when sending myself a message from gmail. For a bit of a change, this is an IPv6 only server, and it has IP 2a00:1450:4864:20::529. So let's look it up with -x:

    delv -x 2a00:1450:4864:20::529
    ; unsigned answer
    9.2.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.ip6.arpa. 43200 IN PTR mail-ed1-x529.google.com.
There are some special domains of the format <reverse-ip-address>.in-addr.arpa and <reverse-ipv6-address-digits>.ip6.arpa that point (PTR) to the DNS name. dig/delv know how to reverse and query appropriately. You can use this for precisely this reason: to look up the intended host. This makes for a nice sanity check during email sending, because in SMTP I can announce myself as any host I like, but the recipient can do a reverse lookup on my IP and see if I'm telling the truth.

One of, but my no means the only, reasons you can't send email from a home connection is because you announce yourself as smtp.zahllos.example but your reverse dns resolves to something like <somegeneratedhostname>.dynamic.residential.isp.com. You can find out yours: type "what is my ip" into google, copy-paste that and do a delv -x on it and see what your "hostname" is according to the internet.