News for hackers. Stuff that matters.
Re:1.1.1.1 outage explanation by gcommer on Friday June 01, 2018 @12:37AM (#17202608)
This is a great write up. It's also why the DNS root servers have a policy of surviving DDoS through massively over-provisioned, multi-org, anycasted redundancy rather this sort of smart DDoS mitigation that drops traffic: DNS is so critical that any risk of dropping real traffic is unacceptable. (obviously, such a scale is impractical for 99% of services)

A good takeaway from this outage for the average user would be to make sure that your fallback DNS resolvers are operated by totally separate providers. (eg, configure 1.1.1.1 with 8.8.8.8 as a fallback, rather than 1.1.1.1 and 1.0.0.1) (Edit: fixed cloudflare's secondary address)

Re:1.1.1.1 outage explanation by crtasm on Friday June 01, 2018 @10:24AM (#17205542)
I read on the Pi-Hole forums that 'fallback' is a misleading term because clients don't work that way - they will happily spread requests between two functioning DNS servers. Can anyone confirm this or provide further insight?
Re:1.1.1.1 outage explanation by mjcl on Friday June 01, 2018 @10:33AM (#17205626)
Different OSes handle it differently. Windows tries the primary, waits 1 second and then starts trying secondaries.
Re:1.1.1.1 outage explanation by dharmab on Friday June 01, 2018 @01:04PM (#17207464)
glibc tries in order. musl (Alpine Linux) tries all in parallel and returns thebl first resolution (If any).
Re:1.1.1.1 outage explanation by cookiecaper on Friday June 01, 2018 @11:24AM (#17206155)
Most systems do not round-robin by default afaik. They just try the listed DNS servers in order (and resolv.conf will only respect the first 3 IIRC).

DNS fallback is often also misunderstood to mean "fall back if the domain is not found", but it really means "fall back if the name server fails to respond". If the domain is not found (i.e. servers returns a valid NXDOMAIN response), most resolvers do not consult any other name servers.

Round-robin, cascading name lookups, and other frequently desired functionality can be obtained through dnsmasq or similar caching/forwarding name servers.

Re:1.1.1.1 outage explanation by LinuxBender on Friday June 01, 2018 @04:01PM (#17209546)
There are also options for libc that affect this behavior that can be set in /etc/resolv.conf like:

    options timeout:2 retries:2 max-inflight:768 max-timeouts:100 single-request-reopen rotate
There are man pages that describe these options.
Re:1.1.1.1 outage explanation by TravelTechGuy on Friday June 01, 2018 @10:59AM (#17205838)
Can confirm. If I have a secondary DNS specified in my router, other than my pi-hole, it becomes useless. My guess is the router is either measuring response time, and goes with the most efficient, or otherwise round-robining the requests. Either way, requests bypass the pi-hole in such quantities that it became useless.

PS: someone here mentioned that this behavior is OS-dependent - nope, this happens on the router level, and all devices in my apartment suffer.

Re:1.1.1.1 outage explanation by jschwartzi on Friday June 01, 2018 @11:34AM (#17206273)
It depends on how your router's DHCP server is configured. If you configure your router to pass its own IP address out as the DNS server for the local subnet then the router's behavior dictates how DNS works. If your router is passing out an external DNS in the DHCP configuration, then you'll get OS-dependent behavior.

My router uses a DNS resolver internally, and it will spread-cast to multiple DNS servers and use the quickest response it can get. It also caches using the TTL in the DNS response, and so it will serve up cached records transparently.

Re:1.1.1.1 outage explanation by detaro on Friday June 01, 2018 @11:01AM (#17205879)
Then your devices are likely using your router as a DNS resolver, which in turn talks to your pihole and the external one. And thus it depends on your router's OS what it does.
Re:1.1.1.1 outage explanation by iforgotpassword on Friday June 01, 2018 @07:06PM (#17211284)
Dnsmasq has different modes if you give it multiple upstream resolvers. By default it only queries the fastest one, which it determines by every now and then sending a query to all the servers and picking the one that replies first. You can tell it to always query all servers, or always query them in the order specified.
Re:1.1.1.1 outage explanation by LinuxBender on Friday June 01, 2018 @03:56PM (#17209477)
If your recursive DNS servers are Unbound DNS, they will keep note of which infra (upstream forwarder) nodes are responding and will blacklist the ones that do not. You can see this with

    unbound-control dump_infra
After the infra ttl expires, they will probe the down nodes. If they respond, then traffic will be distributed among them again.
Re:1.1.1.1 outage explanation by berti on Friday June 01, 2018 @03:49AM (#17203397)
Or 9.9.9.9 if you're not as comfortable with Google services.
Re:1.1.1.1 outage explanation by spondyl on Friday June 01, 2018 @04:32AM (#17203568)
For anyone wondering, 9.9.9.9 is https://www.quad9.net/ who claim to not only resolve requests but also check them against IBM X-Force's threat intelligence database.
Re:1.1.1.1 outage explanation by sp332 on Friday June 01, 2018 @10:00AM (#17205373)
Also from https://www.quad9.net/faq/

Secure IP: 9.9.9.9 Provides: Security blocklist, DNSSEC, No EDNS Client-Subnet sent. If your DNS software requires a Secondary IP address, please use the secure secondary address of 149.112.112.112

Unsecured IP: 9.9.9.10 Provides: No security blocklist, DNSSEC, sends EDNS Client-Subnet. If your DNS software requires a Secondary IP address, please use the unsecured secondary address of 149.112.112.10

Client-Subnet lets providers with widely-distributed servers pick one that's near you to serve your content.

Re:1.1.1.1 outage explanation by nerdponx on Friday June 01, 2018 @09:08AM (#17204908)
For the curious, they also have an extremely detailed privacy policy: https://www.quad9.net/policy/

Seems like a fair deal to me. I get free DNS service, and companies sponsoring this program get metrics on threats and general Internet usage. I'm a little skeptical of their claims that individuals can't be identified from their anonymized data. E.g. I probably only get one or two hits on my personal website every week, so it might not be hard for a malicious employee to deanonymize visitors to my site.

Some highlights from the policy:

Many nations classify IP addresses as Personally-Identifiable Information (PII), and we take a conservative approach in treating IP addresses as PII in all jurisdictions in which our systems reside. Our normal course of data management does not have any IP address information or other PII logged to disk or transmitted out of the location in which the query was received. We may aggregate certain counters to larger network block levels for statistical collection purposes, but those counters do not maintain specific IP address data nor is the format or model of data stored capable of being reverse-engineered to ascertain what specific IP addresses made what queries.

There are exceptions to this storage model: In the event of events or observed behaviors which we deem malicious or anomalous, we may utilize more detailed logging to collect more specific IP address data in the process of normal network defense and mitigation. This collection and transmission off-site will be limited to IP addresses that we determine are involved in the event.

...

We do not correlate or combine information from our logs with any personal information that you have provided Quad9 for other services, or with your specific IP address.

...

Quad9 DNS Services generate and share high level anonymized aggregate statistics including threat metrics on threat type, geolocation, and if available, sector, as well as other vertical metrics including performance metrics on the Quad9 DNS Services (i.e. number of threats blocked, infrastructure uptime) when available with the Quad9 threat intelligence (TI) partners, academic researchers, or the public.

Quad9 DNS Services share anonymized data on specific domains queried (records such as domain, timestamp, geolocation, number of hits, first seen, last seen) with its threat intelligence partners. Quad9 DNS Services also builds, stores, and may share certain DNS data streams which store high level information about domain resolved, query types, result codes, and timestamp. These streams do not contain IP address information of requestor and cannot be correlated to IP address or other PII.

...

Quad9 does not track visitors over time and across third-party websites, and therefore does not respond to Do Not Track signaling.

Re:1.1.1.1 outage explanation by joveian on Friday June 01, 2018 @01:27PM (#17207739)
At least for personal use, OpenNIC is nice and many of the servers say they do not keep logs. I use the 185.121.177.177 (2a05:dfc7:5::53) anycast server and it works well. They are more likely to disappear randomly than the ones run by large companies.

https://servers.opennic.org/

Re:1.1.1.1 outage explanation by alibert on Friday June 01, 2018 @02:02PM (#17208171)
Am I the only one not comfortable using DNS servers running by random volunteers? Is there any "vouching" of the operators or regular checks on common domain on those OpenNIC servers?
Re:1.1.1.1 outage explanation by joveian on Sunday June 03, 2018 @03:46PM (#17221358)
Well, my computer runs a bunch of software written by random volunteers so personally I'm not that worried about it. I personally prefer that to the available alternatives. Yes it would be great to have monitoring (of all the dns services) and I'm not sure if anyone does that, but considering the perpetual tran wreck that is DNSSEC it doesn't really alter anything as all dns is vulnerable.

With https, whoever you end up contacting needs to cough up a valid certificate for the domain in the url. I run https everywhere to try to get that protection as often as possible. In practice there are still ways that dns tricks can cause trouble but they are not as bad as you might think and browsers are slowly pushing an https only web (I hear Chrome will soon start marking all http sites as "insecure" rather than https sites as secure). ssh has its own authentication method and I do try to verify new hosts via another secure chanel.

Speaking of not trusting companies, I am reminded that at one point I noticed that CentryLink seems to be intercepting all dns traffic no matter the intended destination, so without either a secure connection past the ISP or maybe a nonstandard port it may not matter what dns server you try to use. Hopefully all ISPs that do this do the horrible redirect of invalid domains thing so attempting an http connection to an invalid domain might show if this is the case (I found it trying some of the nonstandard domains that OpenNIC resolves).

Re:1.1.1.1 outage explanation by BicyclRepairMan on Friday June 01, 2018 @02:37PM (#17208588)
You're definitely not alone. This sounds a little too Tor-ish for my taste.
Re:1.1.1.1 outage explanation by bluedino on Friday June 01, 2018 @08:10AM (#17204528)
I’m amazed at how many people abandoned other providers and blindly switched to 1.1.1.1

I can’t even use that address with the ISP Alestra in Mexico

Re:1.1.1.1 outage explanation by toomuchtodo on Friday June 01, 2018 @09:36AM (#17205136)
The lack of operational knowledge is real, and there is no easy fix other than resilient defaults by a benevolent dictator (in this case, resolvers across netblock and provider demarcations) or spending the time to educate yourself first on the “why” then the “how”.
Re:1.1.1.1 outage explanation by Dylan16807 on Friday June 01, 2018 @01:27AM (#17202806)
(the cloudflare secondary is 1.0.0.1)
Re:1.1.1.1 outage explanation by cmurf on Friday June 01, 2018 @02:05AM (#17202952)
Cloudfare makes it weirdly difficult to find this. 1.1.1.1 is plastered over many pages but not concomitant with the secondary.
Re:1.1.1.1 outage explanation by chdefrene on Friday June 01, 2018 @02:31AM (#17203057)
That is not my experience at all. By following the `install`-instructions on http://1.1.1.1, all available DNS addresses are listed for both ipv4 and ipv6.
Re:1.1.1.1 outage explanation by Mononokay on Friday June 01, 2018 @05:00AM (#17203701)
FYI, https://1.1.1.1 works over SSL, so there's no need to format it http://1.1.1.1.
Re:1.1.1.1 outage explanation by bobf on Friday June 01, 2018 @02:07AM (#17202960)
That's not necessarily a bad thing, since it's probably better to use another source for a secondary if that's your primary.
Re:1.1.1.1 outage explanation by tracker1 on Friday June 01, 2018 @05:01PM (#17210248)
I'm using cloudflare first, and google as secondary in my router. For a while I was running my own, but it became less necessary when I stopped doing as much dev at home.
Re:1.1.1.1 outage explanation by h1d on Friday June 01, 2018 @08:05AM (#17204498)
Not if you're not comfortable giving up so many of your internet activity to Google but as others have pointed out there are alternatives to spread.