News for hackers. Stuff that matters.
Re:Massive Dyn DNS outage by 6d6b73 on Friday October 21, 2016 @08:17AM (#12759731)
All this talk about redundancy, real-time apps, scalable architecture and and a "simple" DDOS against DNS architecture brings half of the internet down. Honestly did nobody think about having a spare dns at some other company? or even backup dns server exactly for a situation like that?
Re:Massive Dyn DNS outage by mancerayder on Friday October 21, 2016 @12:28PM (#12761785)
If ONLY they had 10X developers ...

EDIT: joking aside, the issue with multiple DNS providers is primarily (in my experience at the company I'm at having investigated this in the past) intelligent DNS entries. Example, 'return these A records, in this order, based on the number of requests, roughly balanced'.

There's no universal standard, just common aspects. DNS Provider A has one set of features, names for returning A records in some way, based on some weighted averaging, provider B has a different mechanism. So as an infrastructure person you have to:

1. Investigate provider A and B features for intelligent DNS (it's not even universally called intelligent DNS!), and mentally parse the commonalities and differences

2. Create a custom mechanism to keep them in sync internally. So you hope that A and B have an API for maintaining the records.

3. Ensure that when someone in your org wants to make an update, A and B update at the same time in the same way

4. APIs don't change.

Re:Massive Dyn DNS outage by dsl on Friday October 21, 2016 @12:56PM (#12762139)
In my experience Dyn pushes this functionality very hard during customer on boarding, effectively as a lock in for their platform.
Re:Massive Dyn DNS outage by mancerayder on Friday October 21, 2016 @01:32PM (#12762569)
Well I really think those features are useful, though. They let you get some powerful de-facto load balancing and HA and other complex 'features' without any new hardware or much complexity. They give you some easy creative freedom in designing a redundant infrastructure, give you some geo-aware DNS the likes of which I remember paying Akamai a lot of money for years go, etc. But the lock-in problem is absolutely right.

I'm also familiar with the aggressiveness of that sales team. I prefer another provider and they were trying to solicit our business by specifically calling out our provider as amenable to a DDOS attack which had occurred.

Someone with a simpler setup with standard DNS features will find it much easier to use multiple providers, of course.

Re:Massive Dyn DNS outage by bhaak on Friday October 21, 2016 @08:22AM (#12759773)
From where I'm looking at the internet (central Europe), I don't notice anything.

Maybe your internet on the other side of the Atlantic is broken, ours seems to be working fine. ;-)

Edit: Looks like the eastern part of the USA is affected: https://cloudharmony.com/status-for-dyn

Re:Massive Dyn DNS outage by a3n on Friday October 21, 2016 @11:37AM (#12761279)
> Maybe your internet on the other side of the Atlantic is broken, ours seems to be working fine. ;-)

Works on my continent.

Re:Massive Dyn DNS outage by tzaman on Friday October 21, 2016 @09:05AM (#12760043)
Not really, I can't access our production servers which are in US east. Can't access Intercom with which we provide customer support. Our clients are mailing us that payment provider doesn't work either. So we're losing money while being in central EU.
Re:Massive Dyn DNS outage by 551199 on Friday October 21, 2016 @12:38PM (#12761893)
I'm in Europe now and worked fine something like 30 min ago.

Now facing similar issues reported in original post.

Re:Massive Dyn DNS outage by pamvegas on Friday October 21, 2016 @03:06PM (#12763567)
I'm in Las Vegas and I haven't had access to several sites all day. I don't think this is limited to eastern U.S.
Re:Massive Dyn DNS outage by xXCrusherXxFTW on Friday October 21, 2016 @04:52PM (#12764652)
Same Im in Illinios and I can't play destiny which is made by bungie which is down for some fucking reason
Re:Massive Dyn DNS outage by planteen on Friday October 21, 2016 @03:33PM (#12763842)
Yep same issue in Denver
Re:Massive Dyn DNS outage by ibjhbrown on Friday October 21, 2016 @02:21PM (#12763126)
Things are down here on the west coast too...
Re:Massive Dyn DNS outage by ljosa on Friday October 21, 2016 @08:22AM (#12759767)
The TTL for the glue records of a .com domain is 48 hours, so even if you have Route53 set up and ready to go, it takes a long time to switch the zone away from Dyn.

We switched from Dyn to Rout53 a few weeks ago. It took about 12 hours before half of the traffic had shifted over.

Re:Massive Dyn DNS outage by 6d6b73 on Friday October 21, 2016 @08:27AM (#12759812)
That's the reason to have your DNS at at least two different companies, working in tandem. In a case where one is down, your Unicorn Corp doesn't go down with it.
Re:Massive Dyn DNS outage by acdha on Friday October 21, 2016 @09:03AM (#12760027)
Exactly: there's nothing wrong with only using one provider if you're not willing to pay for two services but if you can't afford downtime you really need active diversity all the way down.

Route53 uses a bunch of different top-level domains for the same reason – if someone does manage to take the .com servers offline you'll be glad .co.uk is run by a separate organization.

Re:Massive Dyn DNS outage by St-Clock on Friday October 21, 2016 @09:02AM (#12760022)
How does that work in practice? Even if I set NS records pointing to two different DNS providers, I don't think a DNS client would automatically switch and retry if one is too slow to respond/times out.
Re:Massive Dyn DNS outage by acdha on Friday October 21, 2016 @09:08AM (#12760056)
Most DNS resolvers will automatically try each NS record until they get a response. That might be your ISP rather than your iPhone but that's an old practice going back to when the internet was even less reliable because some random Sun box under someone's desk failed.

Modern web browsers will also do the same thing if a query returns multiple A records and they get a connection error.

Re:Massive Dyn DNS outage by 6d6b73 on Friday October 21, 2016 @09:05AM (#12760045)
Why not? That's the whole idea behind having more than one NS server isn't it?
Re:Massive Dyn DNS outage by tuke on Friday October 21, 2016 @10:24AM (#12760685)
N.b., unless things have changed, I don't think Dyn (for one) allows secondary DNS. Maybe that has changed. {?}
Re:Massive Dyn DNS outage by BlackGuyCoding on Friday October 21, 2016 @12:41PM (#12761924)
agreed 100%. This kind of thing shouldn't happen and this widespread. It's like no one is preparing for worse case scenarios until AFTER something tragic happens.

On the West Coast and I just lost twitter/soundcloud/github - 9:40 am PST

Re:Massive Dyn DNS outage by karmajunkie on Friday October 21, 2016 @12:57PM (#12762149)
loss of twitter, soundcloud, and github is what passes for tragic these days?
Re:Massive Dyn DNS outage by oxtoacart on Friday October 21, 2016 @02:34PM (#12763249)
As a programmer, the loss of GitHub is a pretty big deal in terms of work productivity, and as a maintainer of production systems that rely on assets from GitHub, it's a pretty big availability issue.
Re:Massive Dyn DNS outage by karmajunkie on Friday October 21, 2016 @04:31PM (#12764435)
I mean, i get it, its a big deal to us. But tragic? Syria is tragic. Github is just above annoying.
Re:Massive Dyn DNS outage by waterhouse on Friday October 21, 2016 @09:44PM (#12766547)
Let's say that it's tragic for the people whose job it is to maintain those services.
Re:Massive Dyn DNS outage by exolymph on Friday October 21, 2016 @12:58PM (#12762165)
Try PayPal. That's gonna be a big issue for everyone who uses their payment processing.
Re:Massive Dyn DNS outage by sphinx65 on Friday October 21, 2016 @01:04PM (#12762243)
Yeah, as if a "spare DNS" has anything to do with solving the problem. You could get into your own infrastructure, but your customers are still screwed.
Re:Massive Dyn DNS outage by 123456789101112 on Sunday October 23, 2016 @07:25AM (#12773049)
..and they thought war happens only with guns and tanks. don't the know, digital is the new platform to kill each other?
Re:Massive Dyn DNS outage by xtracto on Friday October 21, 2016 @03:45PM (#12763952)
Maybe it is time to widely adopt namecoin ?
Re:Massive Dyn DNS outage by blazinDC on Friday October 21, 2016 @04:41PM (#12764543)
Yes!! For sure, after such an attack it will force the internet into rethinking their DNS strategies. Why commit to having a single point of weakness?

For those interested: https://namecoin.org/

Re:Massive Dyn DNS outage by rotauserch on Friday October 21, 2016 @09:18AM (#12760130)
they need a round robin dns that is geographically dispersed