Universally? Perhaps in the US? But IoT devices are sold worldwide.
While I do run my own recursive resolver I checked my ISP's and they're behaving fairly reasonable and do none of the above and I have a direct (contractual) relationship with them and we reside in the same jurisdiction so at least in principle I could apply pressure to them if they do something shady. The same can't be said about google or cloudflare.
> Just be aware DNS over HTTPS is a thing now, and while the devices I’m responsible for aren’t going to try and evade your redirects, the companies that are trying to make sure ads get delivered will absolutely switch to DoH which will be much more difficult to work around.
I'm sure google had only our best interests in mind when unleashing that on us.
Before going with the conspiratorial take note that Mozilla shipped DoH early and both Microsoft and Apple implemented it. Untrustworthy ISPs are a real problem even if Google deservedly gets suspicion about their motives. This isn’t another AMP.
Here's an article from November 19, 2020 about the rollout and some of the criticisms/backlash:
https://www.zdnet.com/article/fearing-drama-mozilla-opens-pu...
and DoH is absolutely designed to get around network based security and filtering, both for Ad's and other reasons.
You can just say you don’t follow this closely. Mozilla is not perfect but they do push for privacy, with an increasingly limited amount of negotiating power.
> DoH is absolutely designed to get around network based security and filtering, both for Ad's and other reasons.
This is similarly reflecting a poor understanding of the situation. DoH can’t get around network filtering - if you block packets, there’s no magic trick to bypass it. It’s great for preventing ISPs from tampering with traffic or monitoring activity (this will also require eSNI to complete) but it’s not giving an attacker any capability they didn’t already have. If you’re concerned about security you’re fooling yourself if you don’t have endpoint management and some level of network segmentation and egress control. Attackers have hard-coded DNS servers, C&C endpoints, etc. for decades.
The problem is that your relationship with your network devices is logically the same as a totalitarian country's/company's relationship with their citizens/users. So any protocol that prevents censorship/surveillance by ISPs also impinges upon bona fide network administrators. Corporate networks have the same dynamic, although there are few tears shed when it becomes harder for them to tamper with users' traffic.
The right answer is to make sure devices that have any Internet access run code you control. The root of the problem here is buying a "smart" TV, hooking it up to the network, and then expecting to tame all of its user-hostile anti-features by policing its communication. The only way to use black box IoT devices is to remove all general Internet access from them, allowing communication only with hosts you do control. For instance I've got a network of tp-link bulbs that are all controlled from a Home Assistant instance, and they never have and never will get a packet out to the larger Internet.
I agree that the core problem here is blocking egress entirely – and that’d be a good area for home routers to add UI polish so you could easily allow your TV to hit Samsung.com if there’s an update you need before turning it back off. Unfortunately that’s going to be a losing game for many devices and that really hits at the root cause: we need strong regulation controlling privacy because trying to stop a well-funded company with purely technical measures is almost always a losing game.
> hit Samsung.com if there’s an update you need
Why would you need an update? Updates are mainly necessary for security, which you don't need if the device isn't on the Internet. If the device doesn't have all the features you expect out of the box, return it within the return period. There's a small corner case where an update could carry significantly increased functionality, but it seems easier to ad-hoc address that down the line rather than plan for it. Carelessly doing updates is a good way to break your device.
> Unfortunately that’s going to be a losing game for many devices
I don't see how it's a losing game if you play it correctly. Fine grained policing of types of traffic is a losing game, but wholesale denying transit isn't. There is little difference between my network of tp-link bulbs and a local modbus network.
You've never had a problem which was fixed by an update or something which added support for, say, a new model peripheral? I have, which is why I allowed for the possibility of wanting to do this on the schedule of your choosing but not the default case.
> I don't see how it's a losing game if you play it correctly. Fine grained policing of types of traffic is a losing game, but wholesale denying transit isn't. There is little difference between my network of tp-link bulbs and say a local modbus network.
I was thinking less narrowly than devices which never need to be online. A TV connected to other players can run entirely offline but there are many other things which legitimately need connectivity and there's no good way to prevent that. For example, think about a device like a Chromecast or Fire TV, or those Facebook video chat appliances — people buy those to stream content so the most you can do is force the vendor to send marketing stuff through the same endpoint they use for your content, and that's increasingly hard to filter (think how useful a “it goes to an IP in AWS. Block y/n?” prompt is). That's why I said it'll require a legal fix since a large fraction of the most invasive devices either already do or could trivially be modified to mix other data in with the traffic needed to function.
For embedded devices? No. I can imagine it happening in general, but I don't think I would ever buy into a proprietary ecosystem so hard that there would be peripherals, and newly released ones at that. Still I would be cautious about doing said updates, lest they ruin the device I already have. Like I've got a newer Marantz receiver that works great and hasn't seen the Internet in several years. Even if they developed some new desirable feature, why would I want to let it reflash itself and possibly break, or even just get slower (software bloat)? I'd rather just continue using it as I bought it.
> there are many other things which legitimately need connectivity and there's no good way to prevent that
I sort things into categories. A TV would be in the category of "wtf would you ever hook that up online" - Internet access can only enable anti-features. A Chromecast is a different category - single purpose disposable device that if it turns into shit you just throw it out. Ads and surveillance are part of its price, and if your goal is to avoid them, you should just setup a Kodi box and call it a day.
Legally I don't really see what you're getting at here. I can see a law for my TV category, but leaving it disconnected or pulling the 5G modem will also solve that. How would you even begin to solve the Chromecast problem with a law? Maybe in the EU you could convince them to mandate unbundling ads from a service, but in the US exploiting consumers by shoving ads at them is one of the most popular business models. I don't see that ever changing via the legal system.
The solution is to start doing network filtering: if you block packets to unapproved servers, you can actually stop this. You’ll need to run your own proxy, of course, but that’s always been the only way to actually accomplish that goal.
This is a very good point and I am dealing with this myself on my home networks.
Like any household/family we have some number of dubious/untrusted devices that still need Internet access.
By establishing my own recursive resolver I can act as a chokepoint (and monitoring point) for their behavior online. It's a very elegant solution, actually, and I have created a nice integration between my datacenter-hosted resolver and nextdns.io as the adblocking upstream DNS.
DoH breaks all of this.
I have no interest in diving down the "MITM my own network by inserting custom certs into embedded devices that may or may not use them".
Since we're talking about it, though, it occurs to me that you could quickly do a DoH lookup to every single new IP connected to, outbound, from your network - and then block all IPs that answer your DoH query. You're basically pre-testing all new SSL connections to see if they are to a DoH resolver that you (presumably) don't want to talk to ...
This solves the CDN problem ... does it solve the problem entirely ? I have only just thought of this moments ago ...
This only works for the subset of devices which use the local DNS. If they use any of the well-known techniques to avoid that filtering it's completely ineffective.
> Since we're talking about it, though, it occurs to me that you could quickly do a DoH lookup to every single new IP that initiates a new connection, outbound, from your network - and then block all IPs that answer your DoH query.
It doesn't solve the CDN problem: CDNs will route traffic based on the hostname and blocking them will have a degree of collateral damage which most people can't work with. Setting up your own HTTPS proxy avoids this.
If you also block all port 53 after allowing your own resolver ... you may have some headaches with devices that refuse to use the DHCP provided resolvers but you know they aren't going to other resolvers.
That kind of control is what DoH breaks and I'd love to find an elegant (non-MITM proxy) solution for it ...
And in the context of pihole and such, avoiding that means editing the DNS response to remove those public keys. Which takes us full circle back to "do I control DNS for this gadget, or not".
Of course, jailbreaking opens up other security issues, so it goes back to what you can tolerate.
[0] https://checkrain.org/ [1] https://letsencrypt.org/getting-started/
I would imagine you can use this to push any certificate that you can also push to an iOS/iPadOS/macOS device.
I would also love to MitM myself in some cases, mostly because it’d be interesting to see what’s going on.
So before applying that mentality, it would be wise to consider what your experience would be like if all your neighbours, friends, colleagues etc also did that on their networks.
For this reason I wouldn't recommend buying a device like the Chromecast, in which the user can't configure the network settings. Instead maybe consider something like the Amazon Fire Stick which is not as user-hostile.
The usual approach to setting up a firewall is a default of "block everything" and then selectively allow only what is needed.
Most people cheat and only do this on inbound connections, allowing everything on the egress side, because it's easier. But if you want to block your IoT devices from making outbound https connections, you easily can.
There's nothing really new going on here. It's always been possible to tunnel one protocol over another, or use nonstandard ports, and use encryption on the traffic to hide what you're doing.
The point of the original story is "your" meaning a device you own, is ignoring your network controls
it is highly unlikely that the TV manufacturer is going to allow me to install my own custom root certs to inspect their traffic to HTTPS, so yes DoH and other things are a threat to network security, because if the TV become compromised I have limited administrative controls to prevent it other the blocking it completely which is a poor response to the problem
DoH is a solution in search of a problem that can be solves in better more user friendly ways
Standard DNS is unencrypted. DNS-over-HTTP is encrypted. Or DNSSEC or any number of newer standards that secure the DNS lookup. At that point, filtering will require MITM proxies, whether it's for DNS or HTTP or any other protocol.
It's a trade-off with security on the open network meaning harder penetration and control in your internal network. There's no easy answer.
Why do you say it's not authenticated? If they're using the newer standards then that's what it provides. If they're not then there's no issue with network filtering as usual.
It doesn't matter if you're using your ISP's servers, 8.8.8.8, 1.1.1.1, or a custom server you set up on Digital Ocean somewhere: an on-path attacker can forge DNSSEC responses to you. It's a ridiculous situation.
Follow the money .. watch the actions not the words
You’re going to have to make a more substantial argument to get anywhere with this.
As someone who has used (and still use Firefox) continously since around 2005 it surely feels that way sometimes and some of the decisions I see would make much more sense to me if I knew top management was somehow in Googles pocket.
(That said
1. for my workflows I still consider Firefox the best browser.
2. switching would only make it even easier for Google.
3. I always hope something will change and Firefox will become really really great again or someone will fork it.
Consider.
[0] https://en.wikipedia.org/wiki/Mozilla_Corporation#Google
[1] https://www.theverge.com/2020/8/15/21370020/mozilla-google-f...
There was a recent change to facebook picture albums that lazily loads pictures as you scroll. In large albums this grinds to a halt after a few pages and each scroll takes 30 seconds to load the next set of images. Chrome handles this smoothly. For social media I had to switch to chrome.
Firefox is still better for privacy and I use it when I can even though it feels like it's slowing down after each upgrade.
Abrowser (from trisquel) is what firefox should have been. Best fork out there imo.
Ugh, the situation is even worst in countries with censorship laws. For example, in my country, all ISP are required to intercept all DNS requests and filter all requests to any blocked domains found in the government block list. At least they're transparent about which sites are blocked though (the list is publicly available to query or download), but the fact that all DNS requests are intercepted causes various technical issues and some ISP are trying to profit from it by redirecting the blocked query to their own ads-laden landing pages. They even went as far as inspecting http host header as well as randomly injecting scripts on unencrypted http requests.
I was troubleshooting an issue where a company's client computers which connected to the company VPN wouldn't have internet access post-connection.
The problem turned out to be that instead of sending a NXDOMAIN they'd return their ad server and then _always_ send back some JS to show ads.
The company network used PAC files (these are a piece of JS with a single FindProxyForURL() function) via an internal-only URI to steer most requests to our proxies, while keeping some internal.
The problem came about when clients would first start up the OS would attempt to access the internal-only URI before the VPN client finished connecting. In a normal network it'd get nothing and life would carry on. With this problematic ISP they'd get something that /should/ have been a PAC file, but because it was some other piece of JS without FindProxyForURL() it wouldn't work as a PAC file and thus the client wouldn't go to the proxy.
The expiration on this piece of JS was set to some absurd amount of time, so when the client would eventually try hitting our PAC file server again (happens every 20 minutes on Windows) it wouldn't get our file because it thought the garbage one from the ISP was newer. And the ISP updated their hijacking JS more frequently than we updated our PAC file.
There were two possible solutions to this. One, routinely touch the PAC file so it's date was newer than whatever the ISP put out. Or two, set up an external A record for the internal PAC server name to keep the hijacking from working. We went with the second.
Laughably, DoH is from the same gang of A-record squatters that refused to incorporate SRV into HTTP on the (now very evidently spurious) grounds that it could, in some scenarios, require an extra packet, and they couldn't work out how to make it backwards compatible.
Having been comprehensively hijacked by the interests of advertising companies, my view of the HTTP WG has never been lower. And that's a shame because there are some smart people there, tasked with slowly eroding away the last semblance of end-to-end transparency.
Mozilla did more to unleash it than Google. Someone from Mozilla coauthored the RFC for it, and Mozilla had browser support for it first.
The ISP provided DNS in South Korea is absolutely terrible too. Setting 1.1.1.1 or 8.8.8.8 always serves far better experience than the ISP's.
DNS over HTTPS is an abomination sold as snake oil security.
I am very suspicious of the push for https and the like. I feel it is mainly about hiding the payload from me not any third party.
You might control the resolver on your personal computer (for now). You probably don't control it on your phone. You most likely won't control it on your embedded devices.
I think it's insane that devices can effectively be bricked if they can't phone home. It's nothing short of waste, and I think environmental legislation should require device manufacturers to supply ways of disabling or overriding these mechanisms such that devices can continue to operate regardless of whether home servers are blocked or otherwise out of reach, e.g. company goes belly up, censorship etc.
Actually you probably should return such devices as broken.
It was kind of hard to send back a really nice device that I had just opened up and was ready to fly.
Thing is, some companies just use it as a way to fire their customers.
However, they are also playing these sorts of games with other types of devices, where no such justifications exist ( https://www.eevblog.com/forum/eevblab/eevblab-83-dji-pocket-... ) That needs to be answered by returning the product as defective.
I will admit I haven’t done any further investigation, but simply concluded that the gateway at some point started phoning home and if it didn’t receive a response went into some catatonic state. Maybe I’ll dig deeper at some point, time permitting.
If it doesn't see internet it just blocks itself and goes to a screen "Oops I have no internet".
So you can forget about watching movies from your local server using the VLC app as well. Ridiculous.
Hold up. You are claiming that the fact that DoH prevents DNS requests from being visible in cleartext network traffic is a bad thing?
...what? In a world where the choice is between one party (the DNS provider) having access to my DNS requests and everyone on the network including my DNS provider having access to my DNS requests, I'll choose "DNS provider having exclusive access" every single time.
It is when its my network. If they cared about people sniffing they would use DNSSEC, but still use the network DNS server. DNS over HTTPS is just a way for shady companies to hide what they're doing.
Using DoH, especially one served by an advert company is just signing up to be their open book.
The way we solve this is that we assume any knowledge the device has about the outside world can become obsolete, so we do we have a two layer approach.
The bottom layer is that we have a set of semi hard-coded fallback values that are likely to work in the forseeable future. Updating these fallbacks requires an over the air firmware upgrade which isn't a terribly big deal since we regularly upgrade firmware over the air. The goal of these values is to make sure we can get the device online and direct it to somewhere where we can trigger firmware updates.
The second layer is that one or more times per day we ping a config server that sends a packet with configuration data to the unit. This is typically API endpoints etc. The configuration data is essentially a prioritized list of resources, so if one won't respond it will go to the next on the list (while still trying to determine if a higher priority resource becomes available).
Last week we got a chance to see how this failed over beautifully as multiple resources were removed and a fleet of devices just adapted as they should. (The shutdown of these resources were planned, but presented a good opportunity to do a fire drill).
These devices do usually have an UI. Why not provide some options to the user? Let him choose among different types and providers. I'd set mine to use the one provided by DHCP or enter the address of my resolver manually.
In my experience, product design is generally done with a well-meaning attempt to protect the average user from themselves. People who can and do manage their own networks in sophisticated ways are, unfortunately, far less common than people who have no idea that their ISP fscks with DNS lookups.
Personally, I'd bury this setting pretty deep in an advanced-usage-only tab and behind a notice SCREAMING about how using these settings is unsupported. And then tell support staff that they are not obligated to support whatever crazy configurations people cook up for their home networks.
I understand not doing it at all. A few people will complain, but the number of people who will refuse to buy a TV because it doesn't play nice with their pihole is almost certainly too small to register on any material financial statement, and attempting to please them will generally run into some other point they are unwilling to budge on. The number of people who screw up an advanced setting they don't understand will show up in support costs.
The problem of shitty devices on your private network is a different one.
I recently bought a car, and could not find one without a cellular modem and microphones. Removing the modem voids the warranty. The period where you can opt out is mostly over.
For the manufacturer to avoid a warranty claim due to a modification or aftermarket part, they must show that the defect was linked to the part or modification.
They might have to show that it was caused by, but in any case, if your paint fails prematurely, they can’t say your warranty is void because you disabled the cell connection.
Otherwise it is probably mostly equal to driving without registration. It may not be a felony, but it will be fined.
Somewhat more sophisticated would be hardcoding an IP to a server with a REST endpoint that returns the real final IP. (Basically just like what DoH does, but without calling it DoH).
Even more sophisticated would be hiding the final IP on some kind of public web service like Twitter or Github.
[1]: https://github.com/bambenek/block-doh
[2]: https://github.com/Sekhan/TheGreatWall
Edit: I am mostly curious to see what devices/applications it will break.
Many years ago, a old friend of mine purhcased a new Panasonic Smart TV. It was when "Smart TVs" were just becoming a thing.
I hooked her TV up for her; wiring it into the ATT uVerse modem directly. Other devices worked, but this one did not.
After resetting the modem, factory resetting TV, and making sure the ip address on the TV's menu were displaying properly and matched the router's config (they were), as a young naive tech nerd, I just said:
"Looks like they sold you a dud. Thankfully you kept the receipt!. Either way, you still have warranty to get it replaced."
My friend replied: "Shouldn't I call them first before taking it back?"
I said I didn't think it would help, but go ahead.
About 30 mins laters, she was talking to Panasonic tech support and they asked her to manually enter the DNS entries [I believe it was 75.75.. so Comcrap's], and voila, the TV was online again.
We were very happy it was an easy fix; but that day I deftinely did a LOT of reading on DNS servers.
Till this day, DNS entries are something I always check over when troubleshooting (as well as setting my router to Cloudfare's).
I've been running PiHole-like software on my network for a few years now. A couple of years ago, it would block over 40% of traffic consistently. I never saw ads, and it was nice.
In the last year, blocked traffic has dropped to about 15%, and I'm increasingly getting ads on my phone and Chromecast despite tunneling my traffic through my ad-blocked network and blocking Google's DNS at the network level.
Run 'update your block lists' on your Pihole to make sure the lists are being updated correctly.
Conceivably, there's no need to it to even use "real" DNS at all, you could just run a server that responds to queries like "updateserver.ecorp" and save the hassle of even announcing these servers to the public DNS at all.
Also, DoH or not, there are plenty of other ways to ensure that ads get through a DNS filter. For example, a local hosts file could be included in firmware updates and they would just need to make extra effort to ensure that the server IPs didn't change (an elastic IP or load balancer in AWS would be all you need, then it can persist even if the VM has to be deleted).
Netgear, at one point, also decided that hard-coding an IP address (for NTP, not DNS) was the best solution [0].
--
[0]: https://en.wikipedia.org/wiki/NTP_server_misuse_and_abuse#Ne...
As opposed to the IP addresses that your DHCP server told you about.
This is the list I use: https://public-dns.info/nameservers-all.txt
It boggles my mind it's still a problem in some countries. Last time I saw this kind of notice it was in the nineties, and on a web server, definitely not on a client endpoint. People have had unlimited bandwidth for at least a decade now.
https://old.reddit.com/r/homeautomation/comments/k72lzq/sowh...
I can see myself using it as well, maybe even just to see which devices don't use DNS.
Here's another idea: generate a unique IPv6 address per DNS request, route them to the correct destination, filter other IPs. Not really scalable, but usable on small networks/VLANs.
Why wait?
When you are contracted to build something to spec and you don’t build to spec, you don’t make any money. If you push back on the requested spec, the client will leave and go to another company thus leaving you without any money.
What I’m trying to say unless you are writing the spec, you usually have no chance to change anything
Certainly in Europe I’ve not seen anything even close to that.
I’d love to see a good country-by-country survey on ISP DNS to see just how common this manipulation is.
The toxic ISP issue isn't so dramatic where I live, it used to be worse 15-20 years ago. But the solution has always been: if the ISP is messing with you, you just buy your own router and configure it for your network, with a VPN tunnel if necessary. Ignoring DHCP makes this unnecessarily harder.
I'm one of those "idiots" whose been whistling in the wind against encryption of everything and all kinds of security lockdowns and it's because of this sort of thing. The theoretical threat of someone sniffing my traffic is just not a concern to me compared to the very real and increasing threat of handing all control of my computing to centralized user-hostile powers.
And that along with DoH is contributing to making my life a pain in the butt. How exactly do you folks who avoid our DHCP's DNS expect us to comply with legal filtering requirements? Also, what happens when your hard coded DNS servers are shutdown?
I would argue that this is more a problem with the legal requirements than with the equipment - the law(maker) has expectations you can’t reasonably fulfill.
Doesn’t make your situation any better of course, the law is the law even when it’s impossible
Clear text DNS is the ultimate compromise, a gentleman's agreement if you want, that benefits everyone. We can see just enough to filter what we are required to by law on a best-effort basis, but we never see what you are actually doing thanks to the prevalence of TLS. DoH just broke that agreement.
It's a sad example of how a privacy solution like DoH will eventually result in less privacy, at least in some environments. And I'm not even considering how DoH will be the excuse for totalitarian regimes to up their surveillance antics.
I'm damn sure once I have to do the trusted CA path that someone is going to sell a deep packet inspection solution and present it at some conference where someone in charge will hear about it and then it will be off to the races.
Designing a device to connect to something over the internet even if the network it's connected to behaves strangely isn't random or stupid; it's just in conflict with your goals. Incidentally, last time I ran into a network with legally mandated filtering, I checked whether a google image search for "tits" worked. It did.
Nope, it behaves fine. The owner of the network is serving under age kids. Push too far and its white lists only and block all other IP and I'm sure we'll get deep packet inspection forced on us. Some folks have serious problems with Google Images Search, but you can actually deal with that.
I would also say anyone hard coding DNS into a device is just absolutely unprofessional. Its basically a red flag that any filtering the owner of the network doesn't matter to them.
I'm generally inclined to think an "always use this manually-configured DNS" option is desirable in that situation. Of course, many devices may have a financial incentive (ads) to actively resist the network owner's attempts at filtering.
Filtering is inherently adversarial, and I expect a reasonably sophisticated user on your network could find a way to access some proscribed content. I also expect the users of concern on your network are under five years old and that most of them lack advanced knowledge of networking. Is there an established standard for what qualifies as a reliable-enough filter?
Yet that is what internet actually is for a lot of people...
As is DoT :)
I wonder who's the first sponsor for that thing, dns-over-https...
If your hardware isn't resilient to network failures then it absolutely is your fault.
Please, if you have a Pihole, redirect all DNS through it as described in the article. Just be aware DNS over HTTPS is a thing now, and while the devices I’m responsible for aren’t going to try and evade your redirects, the companies that are trying to make sure ads get delivered will absolutely switch to DoH which will be much more difficult to work around.