News for hackers. Stuff that matters.
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by cperciva on Wednesday December 23, 2009 @08:39PM (#1013210)
And people told me I was crazy for hard-coding IP addresses for s3.amazonaws.com and sdb.amazonaws.com into the Tarsnap code... :-)
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by akl on Wednesday December 23, 2009 @09:24PM (#1013265)
You are crazy for doing that - one instance of downtime doesn't justify ignoring all the advantages DNS brings.
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by cperciva on Wednesday December 23, 2009 @09:56PM (#1013304)
When I made that design decision I wasn't considering the possibility of DNS outages at all; I was just thinking in terms of "there's a huge number of places between me and ultradns where someone could insert a spoofed DNS response".
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by jacquesm on Thursday December 24, 2009 @07:43AM (#1013743)
What if amazon gets a new netblock assigned ?

Can you forcibly upgrade the software in that case ?

Re:Amazon goes down; takes S3, Salesforce, Target, and other... by cperciva on Thursday December 24, 2009 @11:55AM (#1014002)
Of course I can upgrade the software; and there's nothing forcible about it. We're talking about code I'm running on my server here...
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by jacquesm on Thursday December 24, 2009 @07:07PM (#1014566)
Ah, my bad, sorry I thought you meant in the tarsnap client. Then I really don't know why people would make a fuss over you hardcoding the ips in there. All you need to do is keep an eye out in case they change them (which you could even automate).
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by cperciva on Thursday December 24, 2009 @07:42PM (#1014605)
All you need to do is keep an eye out in case they change them

Exactly, and that's what I do. (With the caveat that I look for AWS endpoints being taken out of service, not for changes in what DNS tells me.)

Re:Amazon goes down; takes S3, Salesforce, Target, and other... by derefr on Wednesday December 23, 2009 @10:04PM (#1013313)
Are there DNS servers that support versioning? The best solution I could imagine would simply be to set Tarsnap to normally use the current DNS records for S3, but be able to rollback to a valid zone record if they encounter an update that makes the servers stop resolving.
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by cperciva on Wednesday December 23, 2009 @11:35PM (#1013401)
The Tarsnap client only talks to the Tarsnap server, but I do cache that lookup in order to avoid problems with glitchy DNS resolution.

I could have the Tarsnap server cache DNS lookups if I was only worried about working around DNS outages -- but as I said, that wasn't something I was considering at all when I made the decision to eschew DNS.

Re:Amazon goes down; takes S3, Salesforce, Target, and other... by maw on Thursday December 24, 2009 @10:28AM (#1013895)
Some DNS caches will continue to try giving out expired records if no newer ones can be found. Unclear that it's worth it, though.
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by baq on Thursday December 24, 2009 @05:13AM (#1013642)
you can expect such downtimes every Christmas. why? ask yourself how much protection money UltraDNS was told to pay.
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by justinsb on Wednesday December 23, 2009 @08:43PM (#1013214)
Awesome ... was that for security or for redundancy? Did you have a solution for what would happen if the IP address changed?
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by cperciva on Wednesday December 23, 2009 @08:57PM (#1013231)
Mostly security -- DNS is one of the worst offenders when it comes to protocols with security problems, both in terms of protocol issues and bugs in implementations.

Amazon doesn't move endpoints very often, and I round-robin requests to several endpoints; so in the rare cases an AWS endpoint changes I see a slight decrease in Tarsnap performance and can make the Tarsnap server stop using the dead endpoint before any users are likely to notice.

Re:Amazon goes down; takes S3, Salesforce, Target, and other... by esja on Thursday December 24, 2009 @12:53PM (#1014071)
Hmmm... should I trust my backups to generic software vendor X, or to someone who clearly takes nothing for granted... tough choice. Signing up now.
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by justinsb on Wednesday December 23, 2009 @08:32PM (#1013203)
It is DNS. If you put the EC2/S3 address into /etc/hosts, the services work fine. Affecting lots of other big websites as well apparently (target, salesforce) because they all outsource to UltraDNS
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by shaddi on Wednesday December 23, 2009 @08:37PM (#1013208)
NANOG chatter confirming it is an issue with UltraDNS. Seems to be west coast related.

EDIT: Potentially a DOS attack. From NANOG: "We have some DNS providing type customers (not UltraDNS) receiving a few million packets/sec of UDP/53 DoS traffic, starting at about the same time as the UltraDNS problems. No clue if it's related, but it certainly sounds suspicious. :)"

Re:Amazon goes down; takes S3, Salesforce, Target, and other... by aristus on Wednesday December 23, 2009 @08:45PM (#1013220)
I'm very surprised that they rely on a single vendor. But I guess DNS is one of those things you don't think about until it fails.
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by metachor on Thursday December 24, 2009 @10:46AM (#1013918)
The point of using UltraDNS is that it they provide fast "real-time" failover of DNS routing. This is used, for example, for fault tolerance where the IP address responding to a domain name might change due to failure scenarios or load balancing (where a different server is now primary responder to the domain name, maybe located in a different data center or country). Infrastructure-wise, UltraDNS is kind of like the Akami of DNS, instead of content distribution.
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by justinsb on Wednesday December 23, 2009 @08:40PM (#1013213)
I thought DNS was supposed to try backups servers automatically... any DNS experts able to explain what's going on? Some of the ultradns servers are returning (correct) values, others simply not responding.
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by lsc on Wednesday December 23, 2009 @11:47PM (#1013420)
yeah, uh, if you are smart, you have a secondary DNS provider. But that really requires you managing it yourself. the problem was that many people outsource, which usually means going with only one provider. (now, ultradns does have a good setup, they probably aren't a bad choice for a provider, but having only one is just plain stupid.)
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by boredguy8 on Wednesday December 23, 2009 @11:04PM (#1013365)
Never let truth get in the way of a good headline.
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by artagnon on Thursday December 24, 2009 @01:16AM (#1013489)
Haha.. true. What else can we expect from Techcrunch?
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by jseifer on Wednesday December 23, 2009 @10:08PM (#1013315)
It doesn't say in the TC article and I can't really tell from this thread. How long was Amazon actually down?
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by ggrot on Wednesday December 23, 2009 @08:26PM (#1013196)
I wonder how much money amazon loses per minute 2 days before christmas. Ouch.
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by rlpb on Wednesday December 23, 2009 @09:14PM (#1013252)
I don't think that Amazon itself will lose much, since they're well-known enough that most people who fail to reach them will retry later.

I think the small fry using S3 and EC2 will be the ones who are actually hit by this.

Re:Amazon goes down; takes S3, Salesforce, Target, and other... by cookiecaper on Wednesday December 23, 2009 @09:49PM (#1013293)
I don't know, I don't think it'd be as much as 4 - 7 days before Christmas. Most people know that it's too late to order from Amazon or any other online-only store by the night of Dec. 23. It's probably more money than they'd lose normally, but maybe not that significant?

I have no data to back that up, it's pure speculation.

Re:Amazon goes down; takes S3, Salesforce, Target, and other... by alain94040 on Wednesday December 23, 2009 @09:10PM (#1013246)
At $20B annual, assuming Christmas is a big chunk, I'm guessing $2B (10%) in the last two weeks, 2 minutes would then be worth $200,000?

I think my 10% number is too low, so maybe $500K for two minutes? But this is not profit, just revenue.

Re:Amazon goes down; takes S3, Salesforce, Target, and other... by wglb on Wednesday December 23, 2009 @09:18PM (#1013259)
Do you think it might catch up once it is available again? It is a pretty serious shopping destination, and possibly people retry.
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by robryan on Wednesday December 23, 2009 @09:27PM (#1013269)
It really depends on downtime, if it was more than say an hour a lot of people would probably buy elsewhere, small amount of downtime you wouldn't think would effect sales though.
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by notmyname on Wednesday December 23, 2009 @08:52PM (#1013228)
surely they mean "Amazon goes down and takes the Internet with it"
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by slig on Wednesday December 23, 2009 @09:26PM (#1013267)
Only if TC was hosted there.
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by Sam_Odio on Wednesday December 23, 2009 @08:24PM (#1013190)
It looks like a DNS issue, since while http://amazon.com isn't working for me http://72.21.207.65 is (kind of).

EDIT: "No A records were found for amazon.com" http://www.zoneedit.com/lookup.html?host=amazon.com&type...

Re:Amazon goes down; takes S3, Salesforce, Target, and other... by mattiss on Thursday December 24, 2009 @01:55PM (#1014185)
When Rackspace goes down and takes TechCrunch with them, the whole INTERNET goes down. When Amazon goes down, Salesforce, Target, and others...
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by tinio on Wednesday December 23, 2009 @08:44PM (#1013217)
It looks to be back up now.
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by newhouseb on Wednesday December 23, 2009 @09:22PM (#1013263)
Why wouldn't such a company run their own name servers? I understand it's "yet another thing to maintain," but I've set up bind before... didn't seem that bad.
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by evgen on Wednesday December 23, 2009 @10:28PM (#1013336)
This is one of those services that a dedicated provider can sometimes do better than internal IT. Ultradns, for example, has secure secondaries Colocated with large ISPs so you get some good protection against cache poisoning attacks. Everything they do you could do yourself, but it would cost you a lot more than what they charge. (full disclosure: I am a customer and until this evening I was quite happy with their service and reliability)
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by newhouseb on Wednesday December 23, 2009 @11:11PM (#1013378)
"Colocated with large ISPs so you get some good protection against cache poisoning attacks"

Ah yes, that's not something I factored into my equation. Thanks!

Re:Amazon goes down; takes S3, Salesforce, Target, and other... by count on Thursday December 24, 2009 @12:45AM (#1013467)
Amazon IS a large ISP though...
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by andrewvc on Thursday December 24, 2009 @01:35AM (#1013509)
Yep, they are, to all the tenants in AWS datacenters.

My apartment however currently only can talk to Time Warner which talks to Level 3 and then, finally Amazon. I believe what the parent meant was that UltraDNS is colo'd with end-user ISPs like Time Warner, Verizon, etc.

Re:Amazon goes down; takes S3, Salesforce, Target, and other... by newhouseb on Thursday December 24, 2009 @01:57AM (#1013527)
Right, a lot of DNS attacks rely on the attacker replying to a request faster than the actual DNS server and there's no one closer to the end user than ISPs.
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by rbranson on Wednesday December 23, 2009 @10:25PM (#1013331)
It's relatively trivial to outsource. It's one of those services that's easy to measure, quantify, and manage (from an outsourced perspective). There's also a bit more to it then that. The Anycast routing can be quite difficult to setup and maintain. It's virtually useless outside of a very small set of protocols (DNS being one of them), so it wouldn't make sense for Amazon to bring that kind of talent in-house for something like DNS.
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by kierank on Thursday December 24, 2009 @01:19AM (#1013493)
The Anycast routing can be quite difficult to setup and maintain.

Anycast's not particularly difficult to setup and maintain.

Re:Amazon goes down; takes S3, Salesforce, Target, and other... by lsc on Wednesday December 23, 2009 @11:49PM (#1013426)
the mistake was outsourcing to only one provider. it's easy enough to setup a BIND slave elsewhere that automatically transfers the zone from your primary provider.
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by rbranson on Thursday December 24, 2009 @12:08AM (#1013442)
I'm not sure that "setup a BIND slave" is quite that easy for someone with as much traffic as Amazon.com.
Re:Amazon goes down; takes S3, Salesforce, Target, and other... by lsc on Thursday December 24, 2009 @12:35AM (#1013461)
I'm sure they could hire the muscle to do it.

I'm just saying, relying on just one company is usually a bad idea.

Re:Amazon goes down; takes S3, Salesforce, Target, and other... by tybris on Thursday December 24, 2009 @06:44AM (#1013704)
Internet can be so fragile.