News for hackers. Stuff that matters.
Re:Sendgrid is down by adambenayoun on Thursday March 21, 2013 @09:53AM (#5414989)
Sendgrid customer here.

Since many services rely on SMTP providers like Sendgrid, they should have a way to notify customers when their server go down and transactional notifications may be disrupted.

I shouldn't be notified by someone who know we're using Sendgrid and happen to read HN.

Re:Sendgrid is down by rdl on Thursday March 21, 2013 @09:55AM (#5415004)
Yeah -- outage notifications to a service's users should go out, rather than just being pulled.

Also seems like best practice would be to have SPF/etc. entries in place for multiple ESPs, even if you routinely just use one, and be able to switch, for just this reason.

Re:Sendgrid is down by adambenayoun on Thursday March 21, 2013 @10:02AM (#5415053)
We do have SPF entries set up for another provider, the main problem is that some services may need to warm up an IP prior sending lot of emails through a backup provider.

What it means is that basically you should be using 2 providers at the same time and send 50%/50% on each provider in order to keep your IP warms.

While I think big services should account for such scenario, I think 95% of Sendgrid customers are using them to avoid setting up that kind of redundancy.

Main question is should I create the same redundancy for my hosting provider, DNS services, CDN etc...

Re:Sendgrid is down by rdl on Thursday March 21, 2013 @10:26AM (#5415217)
Most of the really heavy CDN users I know do have multiple CDNs constantly in operation, for contract negotiation reasons if nothing else.

DNS is interesting, especially if you do anything location-based. I currently use just CloudFlare, but am not convinced they have enough internal redundancy on DNS, so investigating using Dyn, or self hosting again, or Route53, or some combination. It gets a more complex if you want to use multiple providers doing anything beyond simple DNS though (the DNS protocol itself is totally fine for this, but most of the config management is provider-specific, and using normal dns zone transfers/notifies doesn't really work in this model)

Re:Sendgrid is down by kstrauser on Thursday March 21, 2013 @10:16AM (#5415137)
What do you mean by "to warm up an IP"?
Re:Sendgrid is down by jrs235 on Thursday March 21, 2013 @10:36AM (#5415290)
If an IP address does not have a history of sending [large amounts of] email it is considered "cold". If that IP address then suddenly begins sending huge numbers of emails, many providers will assume the worst (that the email is being used for SPAM) and either block the IP or mark messages coming from it as SPAM. "Warming up" the IP means sending a smaller number of emails (per time period) and slowly increasing the volume so that Email providers can properly "score" the IP as legit and okay in their systems.

Update: For some reason I couldn't reply to your response kstrauser so here is my reply: That works if and when their is trust. First you have trust the buyer isn't going to abuse the system and wreck the IP [reputation] and you have to trust the buyer isn't giving you stolen info. Obviously things one can work through but again, it involves trust. Also the provider has to have warmed up IPs to give out. Which, having warmed up IPs would actually be a great valued add upsell for those that need them!

Update 2: Thank you FfejL and symfoniq for explaining it better.

Re:Sendgrid is down by kstrauser on Thursday March 21, 2013 @10:48AM (#5415363)
That's when you physically call your sales rep and explain why your using their services and ask them to whitelist you. I say this from experience. If you called my employer (and any of our competitors, I'm certain) and said "I need to host a mail relay that's ready to go from no traffic to tens of thousands of emails per second, here's my company website, and here's our credit card", we'd be happy to make it work for you.

I'm not saying this to advertise but to offer a suggestion: call someone and ask for help. This isn't an unusual need at all and any reputable provider will be quick to help.

Short version: you don't have to "warm up an IP" if you do a little advance homework.

Re:Sendgrid is down by FfejL on Thursday March 21, 2013 @10:55AM (#5415416)
That's not what 'warming up an IP' means.

If Google (or any other ISP) suddenly sees an big spike in email from @example.com on an IP that @example.com hasn't used previously, Google is much more likely to mark those emails as Spam.

So senders need to 'warm up' an IP by sending a small amount of email first, usually for a few days at least.

So if you're a SendGrid customer expecting to send 2,000,000 emails today, you can't just switch to a new ESP and send those same emails. Spam rates will go through the roof.

Re:Sendgrid is down by symfoniq on Thursday March 21, 2013 @10:54AM (#5415410)
jrs235 is referring to IP reputation as measured on the receiving end of the email. In other words, Yahoo or Hotmail might reject your emails if they see unusual activity from an IP address. Until an IP is "established" as having sent a lot of email with low bounce and spam rates, getting bulk email delivered from that IP address will be a frustrating experience.
Re:Sendgrid is down by kstrauser on Thursday March 21, 2013 @11:05AM (#5415499)
Thanks for the clarification! I'll pass that along to our products department. :-)
Re:Sendgrid is down by jrs235 on Thursday March 21, 2013 @11:07AM (#5415520)
Who do you work for?
Re:Sendgrid is down by kstrauser on Thursday March 21, 2013 @11:46AM (#5415877)
Mail me at [email protected] . I'm not trying to be evasive but I like to keep work and personal separate, especially since this story has become more political than I was expecting. I just came in to read about a service outage.
Re:Sendgrid is down by mryan on Thursday March 21, 2013 @09:57AM (#5415016)
They've been tweeting about it for an hour or so: https://twitter.com/SendGrid

I would expect them to send emails to customers in the event of serious issues though, expecting your customers to learn about problems through Twitter isn't that much better than learning about them from HN.

Re:Sendgrid is down by adambenayoun on Thursday March 21, 2013 @10:05AM (#5415073)
Critical notification should be sent to me and not pulled by me. Nice addition would be to have an addon to newrelic that would basically poll services/API like sendgrid and allow you to setup notifications for failure or degradation.
Re:Sendgrid is down by nwh on Thursday March 21, 2013 @09:58AM (#5415027)
> I would expect them to send emails
Re:Sendgrid is down by pytrin on Thursday March 21, 2013 @10:01AM (#5415045)
Preferably not with their own SMTP service ;)
Re:Sendgrid is down by whaevr on Thursday March 21, 2013 @09:57AM (#5415020)
Aaand the PR slide for SendGrid continues. Perfectly valid point though.
Re:Sendgrid is down by acdha on Thursday March 21, 2013 @10:29AM (#5415240)
This is a “PR slide” in the same sense that torching someone's store is PR.
Re:Sendgrid is down by whaevr on Thursday March 21, 2013 @10:35AM (#5415288)
"I shouldn't be notified by someone who know we're using Sendgrid and happen to read HN."

Was what I was replying to, not insinuating the fact that SendGrid's website going down is a "PR Slide"

Re:Sendgrid is down by jlebrech on Thursday March 21, 2013 @10:12AM (#5415112)
how does one send outage notices if the mail is down?
Re:Sendgrid is down by kstrauser on Thursday March 21, 2013 @10:18AM (#5415152)
Have an offsite cloud server with a mail daemon that's ready to send a few thousand messages to customers when your main server is down.
Re:Sendgrid is down by rdl on Thursday March 21, 2013 @01:56PM (#5417385)
It might actually be worthwhile for a third-party status site/monitoring service to have a way to send this mail to users. It would be thousands of users per developer tool, but not millions like consumer services, so technically a much simpler service.

Letting users set "I must ack notifications of outages or keep pinging me" would make sense, too.

I'd probably consider mobile push notifications with confirmation and/or voice or SMS as well.

Basically like Nagios, but for third-party components.