I see it as Protonmail explaining to their users that the failure is not on their end and why they can't do much about a remedy. Seems purely factual. A cheap shot would be generalizing from the event, but I don't see them doing that.

I think I got this completely wrong. What you and other responses are saying makes sense.

Being down is okay. Returning an error message that results in the data being thrown away instead of being requeued is not. Block incoming smtp connections until your app layer is fixed.

> Block incoming smtp connections until you app layer is fixed.

Or returning one of the 4xx status codes which indicate less-permanent failure state like:

- 451 Requested action aborted: local error in processing

Which is kinda like a HTTP internal server error as it can mean anything.

For my comment’s purposes, I assume if this was possible with a flag or config setting (and the code path existed), it would’ve already been done. Doesn’t seem like they can, so they should’ve pulled the handbrake and gone “full stop” without throwing everyone’s mail away (hence blocking incoming connections and let the mail sit in all of the external MTA queues).

Another option would’ve been to accept everything with a very lightweight smtp ingest service, journal it all, and play it back to the full frontend after their code fix was pushed out.

Not an SRE so ¯\_(ツ)_/¯ just some thoughts from my time in a similar role and similar pain points (but thankfully not at this scale)

Yeah, this is a particularly pernicious failure given how email works. Many mailing providers will just mark these as blacklisted, now, and lots of unsophisticated users likely won't notice.

I consider myself sophisticated enough, but my Bitwarden has 700 accounts, of which ~30% old ones are registered with a gmail address, and the rest are handled behind g suite. Granted that last bit might be partly my fault, even though I paid for it. But even for a "sophisticated" user, I have no easy way of knowing if any of these accounts have silently failed to function now, other than by the passage of time and eventually finding out.

Oh, absolutely, even for sophisticated users mitigating may be difficult or impossible depending on exactly what bounced and how. But you at least are aware that this happened, and that you have a problem. Think how many people are out there with no clue what this error meant, or that it signaled an ecosystem problem, or that just had hundreds or thousands of emails silently bounce and unsubscribe.

A lot of people and companies use Gmail. Email providers are definitely getting support requests from users that don't know what's going on.

This is not a cheap shot, but a message to inform users that it's an issue with Google that Protonmail can do nothing about.

More like "if mail to gmail fails it's not us so pleas don't flood the support with comaplains".

> Running a datacenter is no easy task.

Sure, but then there are very view companies which have more experience with running data-centers and (normally) providing reliable email service.

So any outage for more then just a short time is very unusual. I'm really interested what went wrong.