News for hackers. Stuff that matters.
Re:Cloudflare DNS problem by packetized on Monday July 06, 2015 @05:42PM (#9842374)
Dyn also appears to be having a real, real bad day. Glue records vanished like a fart in the wind.

  $ dig @8.8.8.8 ns2.p29.dynect.net

  ; <<>> DiG 9.7.3-P3-RedHat-9.7.3-8.P3.el6 <<>> @8.8.8.8 ns2.p29.dynect.net
  ; (1 server found)
  ;; global options: +cmd
  ;; Got answer:
  ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 2690
  ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

  ;; QUESTION SECTION:
  ;ns2.p29.dynect.net.		IN	A

  ;; AUTHORITY SECTION:
  dynect.net.		600	IN	SOA	ns0.dynamicnetworkservices.net. hostmaster.dyndns.com. 2015070600 10800 1800 604800 1800

  ;; Query time: 2 msec
  ;; SERVER: 8.8.8.8#53(8.8.8.8)
  ;; WHEN: Mon Jul  6 21:42:48 2015
  ;; MSG SIZE  rcvd: 120
Re:Cloudflare DNS problem by mattbeckman on Monday July 06, 2015 @05:53PM (#9842424)
Does CloudFlare use Dyn or are these unrelated issues?
Re:Cloudflare DNS problem by rdl on Monday July 06, 2015 @05:57PM (#9842447)
The affected zones are those cnamed from CloudFlare to Dyn.
Re:Cloudflare DNS problem by sajal83 on Monday July 06, 2015 @06:27PM (#9842566)
Any example hostname thats configured on Dyn and is not resolving?
Re:Cloudflare DNS problem by rdl on Monday July 06, 2015 @06:31PM (#9842589)
Sorry about this: just posted a blog post. Appears to be a Dyn issue, but affects a lot of our customers (including support.cloudflare.com which is Zendesk)

https://blog.cloudflare.com/the-internet-is-a-cooperative-sy...

Re:Cloudflare DNS problem by jgrahamc on Monday July 06, 2015 @06:22PM (#9842540)
This appears to not be a problem with CloudFlare, but a problem with Dyn: https://www.dynstatus.com/. Of course, Dyn and CloudFlare have mutual customers and so if a Dyn has a problem it's possible for a mutual customer to have a problem.
Re:Cloudflare DNS problem by stockliasteroid on Monday July 06, 2015 @06:33PM (#9842596)
Strange, we are using CNAME apex resolution with cloudflare to point to Heroku apps, so I'm not aware that Dyn is used anywhere in that chain, but resolution is still down for those apps.

edit: resolved now.

Re:Cloudflare DNS problem by jgrahamc on Monday July 06, 2015 @06:39PM (#9842610)

    $ dig +short NS heroku.com    
    ns1.p19.dynect.net.
    ns4.p19.dynect.net.
    ns3.p19.dynect.net.
    ns2.p19.dynect.net.
Re:Cloudflare DNS problem by stockliasteroid on Monday July 06, 2015 @06:53PM (#9842669)
Thanks!
Re:Cloudflare DNS problem by hosay123 on Monday July 06, 2015 @06:30PM (#9842579)
Looks like your support site might need a little.. support http://i.imgur.com/X76QdaC.png

You might want to try putting it behind a CDN to cope with the load

Re:Cloudflare DNS problem by jgrahamc on Monday July 06, 2015 @07:07PM (#9842723)
We use Zendesk. Zendesk uses Dyn. Boom!
Re:Cloudflare DNS problem by sinak on Monday July 06, 2015 @05:38PM (#9842347)
More details here on Cloudflare's status page here: http://www.cloudflarestatus.com/incidents/m8hf8q8vs81d
Re:Cloudflare DNS problem by rdl on Monday July 06, 2015 @05:52PM (#9842413)
Sorry -- our ops and engineering teams are looking into this and will update/resolve as quickly as possible.

https://www.cloudflarestatus.com/incidents/m8hf8q8vs81d

Re:Cloudflare DNS problem by iaw on Monday July 06, 2015 @05:54PM (#9842429)
Good luck, this is really rough. I encountered a 101 on a random site before I saw this and was a bit baffled.
Re:Cloudflare DNS problem by kelseydh on Monday July 06, 2015 @05:43PM (#9842376)
I love how even https://support.cloudflare.com/ is down by this.
Re:Cloudflare DNS problem by colinbartlett on Monday July 06, 2015 @05:52PM (#9842417)
They do at least use a different DNS provider for their status page: http://www.cloudflarestatus.com
Re:Cloudflare DNS problem by Someone1234 on Monday July 06, 2015 @06:06PM (#9842481)
I do love how everything is operational...

I see the big blurb about DNS at the top. But it still feels like the table is misleading. When they're having a major DNS outage and every site is "operational" it feels misleading.

Although if this was Amazon's AWS console we wouldn't even have the blurb at the top...

Re:Cloudflare DNS problem by colinbartlett on Monday July 06, 2015 @06:17PM (#9842525)
It's frustrating for me, too. Because my side project https://statusgator.io pulls in this status data and sends alerts to subscribers when services go down.

But since CloudFlare has not indicated any downtime, everything still appears up.

Re:Cloudflare DNS problem by jonny_eh on Monday July 06, 2015 @05:45PM (#9842387)
Makes me feel good as an impacted user. I know that the problem won't be ignored!
Re:Cloudflare DNS problem by xxdesmus on Monday July 06, 2015 @07:20PM (#9842766)
ZenDesk uses Dyn. Hence, it was down. Sorry about that.
Re:Cloudflare DNS problem by kelseydh on Monday July 06, 2015 @06:01PM (#9842462)
If you're on Heroku, one immediate way around this problem is to guide your users towards your heroku domain. I've been doing this for clients contacting me who absolutely need access right this moment.

E.g. https://myappname.herokuapp.com

Re:Cloudflare DNS problem by Someone1234 on Monday July 06, 2015 @06:13PM (#9842512)
That's a nice tip.

Can I ask a question while we wait for Cloudflare to fix this... Heroku looks nice. But also quite expensive after you add on dyno, Postgre/Redis, and "addons." What about Heroku do you like/appreciate that justifies the slightly higher costs? Or do you just like that all of the devops-style stuff is handled by them (as opposed to AWS/Azure/Linode, etc where you'd have to do it yourself).

I am genuinely asking, I have no reason to dislike Heroku, and I know it is quite popular. Just trying to understand what the "value add" is over other cloud services (or why they're dissimilar).

Please don't feel obliged to answer, but if you have the time it would be appreciated/interesting.

Re:Cloudflare DNS problem by kelseydh on Monday July 06, 2015 @06:50PM (#9842661)
No problem, here's my two (relatively unorganized) two cents:

With Heroku (frankly also thanks to Ruby on Rails) I can be a sys-admin, developer, database administrator and QA all at once; far easier than before. For the price of $100k I have seen developers build successful products that used to cost $1M+ to develop.

On Heroku you definitely pay for convenience and ease of use, knowing that you get a benefit in doing so by being able to focus on the development of your app, the part that matters. The costs grow if you get traffic, but through careful management (such as packing multiple processes into one dyno), you can reduce costs. Even if many find Heroku's prices high, they are still much lower than what you see in enterprise land, or even on AWS if you aren't careful.

Heroku's documentation is also incredibly good, and this helps you a lot to move fast. Almost by accident, simply by following Heroku documentation I have found that developers end up following good cloud server practices almost by default (i.e. because Heroku forces you to build your app to follow certain conventions, conventions needed for making your app horizontally scaleable; such as delegating clock processes to a separate process, and not allowing dynos to manipulate or rely on unique file memory specific only to them).

If I architectured or configured my own servers it would have taken 10x as long, exposed me to far more security issues, and I would have probably screwed it up any way. The lesson from this is that I think as developers we need far less freedom to configure than we think we need.

So in conclusion, if you're a small to medium-sized app that does not have complex server or bandwidth needs (i.e. you are not Imgur or Pied Piper), then I highly recommend it. I'm sure many Oracle salesman will be unhappy with this opinion, but take my word for it that the vast majority of applications do not need complex, overengineered server systems to be successful. (After all as a startup, you should not be wasting months planning on how you will scale to support 10 million concurrent users. I've seen that hubris distract failing startups before.)

Re:Cloudflare DNS problem by toomuchtodo on Monday July 06, 2015 @07:10PM (#9842731)
> With Heroku (frankly also thanks to Ruby on Rails) I can be a sys-admin, developer, database administrator and QA all at once; far easier than before. For the price of $100k I have seen developers build successful products that used to cost $1M+ to develop.

Unfortunately, Heroku is down constantly. Don't take my word for it; google "Heroku down" and see how often it occurs.

Microsites? Sure. Hobby sites? Sure. Business applications that generate revenue? Good luck.

Re:Cloudflare DNS problem by kelseydh on Tuesday July 07, 2015 @09:13AM (#9845170)
Heroku is just a layer on top of AWS. I know a ton of successful business applications who rely on Heroku + AWS. Obviously after a certain size you will want to customize your hardware more, but in the early stages Heroku can take you a very far way.

Ello is a recent app that had to scale to millions of users extremely quickly and did on Heroku. Read their technical write up: http://mikepackdev.com/blog_posts/40-5-early-lessons-from-ra...

Ello had issues, and Heroku would be considered amateur for what they were doing, but if you read it Heroku was not the cause of their performance issues -- their application code was. IMO most business applications run into problems due to performance issues within their application code, not because of server downtime.

Regarding how common downtime on Heroku is... yes, Heroku has lots of minor incidents impairing its toolkit and some services, but these are not issues that go notices. Two years in I still haven't had significant downtime from Heroku itself (more than 5 minutes) during business hours where it matters. Cloudflare, and other DNS providers, have frankly given me more trouble.

Re:Cloudflare DNS problem by Someone1234 on Tuesday July 07, 2015 @09:57AM (#9845435)
Thank you for the reply. I appreciate the insight.
Re:Cloudflare DNS problem by kelseydh on Monday July 06, 2015 @05:56PM (#9842436)
Lately my number one bloody cause of downtime has been DNS lookup failures. I switched to Cloudflare hoping to avoid this. If I can't trust Cloudflare, then who the heck can I trust?

Also, is there any way to actually resolve this issue quickly while Cloudflare deals with their fuckup? I don't like being at the mercy of their support team while my site has literal downtime.

Re:Cloudflare DNS problem by troydavis on Monday July 06, 2015 @07:14PM (#9842748)
We had the same question and concluded that the only reasonable way to say more than "Well, we hope this doesn't happen again" was using 2 different authoritative DNS providers (both anycasted). Details: http://blog.papertrailapp.com/dns-outage-on-monday-december-...

It's not a question of trust; at some point, any single provider will get DDoSed or have an operational bug.

The only question is whether you're willing to accept that. If you are, then try to pick one great provider.

If you're not, and are willing to spend a little time to avoid this problem, try to pick two great providers. Since then, DNSimple has made primary/secondary providers even easier: https://support.dnsimple.com/articles/secondary-dns/

Basically, if you don't want this problem, don't pick one provider. It's not bulletproof, but it's a lot closer.

Re:Cloudflare DNS problem by colinbartlett on Monday July 06, 2015 @06:05PM (#9842476)
I use DNSimple and they have a feature that lets you sync to another provider.

I have not used it but, as I understand it, the feature helps you setup slave servers which respond to Notify messages from DNSimple. That way, if they go down, your slave provider will keep things up.

Re:Cloudflare DNS problem by toomuchtodo on Monday July 06, 2015 @07:12PM (#9842741)
> Also, is there any way to actually resolve this issue quickly while Cloudflare deals with their fuckup?

Learn basic troubleshooting so you can determine which vendor is the problem (ie not Cloudflare in this case).

Re:Cloudflare DNS problem by rdl on Monday July 06, 2015 @05:57PM (#9842440)
We should have it fixed very shortly; 99.9% on the cause.
Re:Cloudflare DNS problem by rdl on Monday July 06, 2015 @06:47PM (#9842645)
In this case you would have been unaffected if you used CloudFlare for DNS, rather than using Dyn for DNS and CNAME from CloudFlare to it. Using CloudFlare for DNS is our preferred configuration.
Re:Cloudflare DNS problem by laumars on Monday July 06, 2015 @06:10PM (#9842495)
You could always run your own DNS server. Of course, you'd then lose the other services Cloudflare offer like basic DDoS protection, but that's the trade off you'd have to consider.
Re:Cloudflare DNS problem by elmin on Monday July 06, 2015 @06:14PM (#9842513)
I've never had any trouble with Amazon Route53.
Re:Cloudflare DNS problem by zrail on Monday July 06, 2015 @07:21PM (#9842770)
That's because Route53 is basically bullet proof, such that their SLA guarantees 100% availability. I have switched everything I touch to Route53 and have had zero issues.