Re:Amazon goes down; takes S3, Salesforce, Target, and other...
by cperciva on Wednesday December 23, 2009 @09:56PM
(#1013304)
When I made that design decision I wasn't considering the possibility of DNS outages at all; I was just thinking in terms of "there's a huge number of places between me and ultradns where someone could insert a spoofed DNS response".
Re:Amazon goes down; takes S3, Salesforce, Target, and other...
by jacquesm on Thursday December 24, 2009 @07:07PM
(#1014566)
Ah, my bad, sorry I thought you meant in the tarsnap client. Then I really don't know why people would make a fuss over you hardcoding the ips in there. All you need to do is keep an eye out in case they change them (which you could even automate).
Re:Amazon goes down; takes S3, Salesforce, Target, and other...
by derefr on Wednesday December 23, 2009 @10:04PM
(#1013313)
Are there DNS servers that support versioning? The best solution I could imagine would simply be to set Tarsnap to normally use the current DNS records for S3, but be able to rollback to a valid zone record if they encounter an update that makes the servers stop resolving.
Re:Amazon goes down; takes S3, Salesforce, Target, and other...
by cperciva on Wednesday December 23, 2009 @11:35PM
(#1013401)
The Tarsnap client only talks to the Tarsnap server, but I do cache that lookup in order to avoid problems with glitchy DNS resolution.
I could have the Tarsnap server cache DNS lookups if I was only worried about working around DNS outages -- but as I said, that wasn't something I was considering at all when I made the decision to eschew DNS.
Re:Amazon goes down; takes S3, Salesforce, Target, and other...
by cperciva on Wednesday December 23, 2009 @08:57PM
(#1013231)
Mostly security -- DNS is one of the worst offenders when it comes to protocols with security problems, both in terms of protocol issues and bugs in implementations.
Amazon doesn't move endpoints very often, and I round-robin requests to several endpoints; so in the rare cases an AWS endpoint changes I see a slight decrease in Tarsnap performance and can make the Tarsnap server stop using the dead endpoint before any users are likely to notice.
Re:Amazon goes down; takes S3, Salesforce, Target, and other...
by justinsb on Wednesday December 23, 2009 @08:32PM
(#1013203)
It is DNS. If you put the EC2/S3 address into /etc/hosts, the services work fine. Affecting lots of other big websites as well apparently (target, salesforce) because they all outsource to UltraDNS
Re:Amazon goes down; takes S3, Salesforce, Target, and other...
by shaddi on Wednesday December 23, 2009 @08:37PM
(#1013208)
NANOG chatter confirming it is an issue with UltraDNS. Seems to be west coast related.
EDIT: Potentially a DOS attack. From NANOG:
"We have some DNS providing type customers (not UltraDNS) receiving a few million packets/sec of UDP/53 DoS traffic, starting at about the same time as the UltraDNS problems. No clue if it's related, but it certainly sounds suspicious. :)"
Re:Amazon goes down; takes S3, Salesforce, Target, and other...
by metachor on Thursday December 24, 2009 @10:46AM
(#1013918)
The point of using UltraDNS is that it they provide fast "real-time" failover of DNS routing. This is used, for example, for fault tolerance where the IP address responding to a domain name might change due to failure scenarios or load balancing (where a different server is now primary responder to the domain name, maybe located in a different data center or country). Infrastructure-wise, UltraDNS is kind of like the Akami of DNS, instead of content distribution.
Re:Amazon goes down; takes S3, Salesforce, Target, and other...
by justinsb on Wednesday December 23, 2009 @08:40PM
(#1013213)
I thought DNS was supposed to try backups servers automatically... any DNS experts able to explain what's going on? Some of the ultradns servers are returning (correct) values, others simply not responding.
Re:Amazon goes down; takes S3, Salesforce, Target, and other...
by lsc on Wednesday December 23, 2009 @11:47PM
(#1013420)
yeah, uh, if you are smart, you have a secondary DNS provider. But that really requires you managing it yourself. the problem was that many people outsource, which usually means going with only one provider. (now, ultradns does have a good setup, they probably aren't a bad choice for a provider, but having only one is just plain stupid.)
Re:Amazon goes down; takes S3, Salesforce, Target, and other...
by cookiecaper on Wednesday December 23, 2009 @09:49PM
(#1013293)
I don't know, I don't think it'd be as much as 4 - 7 days before Christmas. Most people know that it's too late to order from Amazon or any other online-only store by the night of Dec. 23. It's probably more money than they'd lose normally, but maybe not that significant?
I have no data to back that up, it's pure speculation.
Re:Amazon goes down; takes S3, Salesforce, Target, and other...
by robryan on Wednesday December 23, 2009 @09:27PM
(#1013269)
It really depends on downtime, if it was more than say an hour a lot of people would probably buy elsewhere, small amount of downtime you wouldn't think would effect sales though.
Re:Amazon goes down; takes S3, Salesforce, Target, and other...
by newhouseb on Wednesday December 23, 2009 @09:22PM
(#1013263)
Why wouldn't such a company run their own name servers? I understand it's "yet another thing to maintain," but I've set up bind before... didn't seem that bad.
Re:Amazon goes down; takes S3, Salesforce, Target, and other...
by evgen on Wednesday December 23, 2009 @10:28PM
(#1013336)
This is one of those services that a dedicated provider can sometimes do better than internal IT. Ultradns, for example, has secure secondaries Colocated with large ISPs so you get some good protection against cache poisoning attacks. Everything they do you could do yourself, but it would cost you a lot more than what they charge. (full disclosure: I am a customer and until this evening I was quite happy with their service and reliability)
Re:Amazon goes down; takes S3, Salesforce, Target, and other...
by andrewvc on Thursday December 24, 2009 @01:35AM
(#1013509)
Yep, they are, to all the tenants in AWS datacenters.
My apartment however currently only can talk to Time Warner which talks to Level 3 and then, finally Amazon. I believe what the parent meant was that UltraDNS is colo'd with end-user ISPs like Time Warner, Verizon, etc.
Re:Amazon goes down; takes S3, Salesforce, Target, and other...
by newhouseb on Thursday December 24, 2009 @01:57AM
(#1013527)
Right, a lot of DNS attacks rely on the attacker replying to a request faster than the actual DNS server and there's no one closer to the end user than ISPs.
Re:Amazon goes down; takes S3, Salesforce, Target, and other...
by rbranson on Wednesday December 23, 2009 @10:25PM
(#1013331)
It's relatively trivial to outsource. It's one of those services that's easy to measure, quantify, and manage (from an outsourced perspective). There's also a bit more to it then that. The Anycast routing can be quite difficult to setup and maintain. It's virtually useless outside of a very small set of protocols (DNS being one of them), so it wouldn't make sense for Amazon to bring that kind of talent in-house for something like DNS.
Re:Amazon goes down; takes S3, Salesforce, Target, and other...
by lsc on Wednesday December 23, 2009 @11:49PM
(#1013426)
the mistake was outsourcing to only one provider. it's easy enough to setup a BIND slave elsewhere that automatically transfers the zone from your primary provider.
Can you forcibly upgrade the software in that case ?
Exactly, and that's what I do. (With the caveat that I look for AWS endpoints being taken out of service, not for changes in what DNS tells me.)
I could have the Tarsnap server cache DNS lookups if I was only worried about working around DNS outages -- but as I said, that wasn't something I was considering at all when I made the decision to eschew DNS.
Amazon doesn't move endpoints very often, and I round-robin requests to several endpoints; so in the rare cases an AWS endpoint changes I see a slight decrease in Tarsnap performance and can make the Tarsnap server stop using the dead endpoint before any users are likely to notice.
EDIT: Potentially a DOS attack. From NANOG: "We have some DNS providing type customers (not UltraDNS) receiving a few million packets/sec of UDP/53 DoS traffic, starting at about the same time as the UltraDNS problems. No clue if it's related, but it certainly sounds suspicious. :)"
I think the small fry using S3 and EC2 will be the ones who are actually hit by this.
I have no data to back that up, it's pure speculation.
I think my 10% number is too low, so maybe $500K for two minutes? But this is not profit, just revenue.
EDIT: "No A records were found for amazon.com" http://www.zoneedit.com/lookup.html?host=amazon.com&type...
Ah yes, that's not something I factored into my equation. Thanks!
My apartment however currently only can talk to Time Warner which talks to Level 3 and then, finally Amazon. I believe what the parent meant was that UltraDNS is colo'd with end-user ISPs like Time Warner, Verizon, etc.
Anycast's not particularly difficult to setup and maintain.
I'm just saying, relying on just one company is usually a bad idea.