News for hackers. Stuff that matters.
Re:Ask HN: What do you self-host? by ricardbejarano on Sunday October 13, 2019 @07:41AM (#21239870)
There you go!

Essentially, this setup achieves 5 features I wanted my DNS to have:

- Confidentiality: from my ISP; and from anyone listening to the air for plain-text DNS questions when I'm on public WiFi. Solution: DNS-over-TLS[1]

- Integrity: of the answers I get. Solution: DNS-over-TLS authenticates the server

- Privacy: from web trackers, ads, etc. Solution: domain name blacklist

- Speed: as in, fast resolution times. Solution: caching and cache prefetching[2]

- Observability: my previous DNS was Dnsmasq[3], AFAIK Dnsmasq doesn't log requests, only gives a couple stats[4], etc. Solution: a Prometheus endpoint

CoreDNS ticks all of the above, and a couple others I found interesting to have.

To set it up, I wrote my own (better) CoreDNS Docker image[7] to run on my Kubernetes cluster; mounted my Corefile[8] and my certificates as volumes, and exposed it via a Kubernetes Service.

The Corefile[8] essentially sets up CoreDNS to:

- Log all requests and errors

- Forward DNS questions to Cloudflare's DNS-over-TLS servers

- Cache questions for min(TTL, 24h), prefetching any domains requested more than 5 times over the last 10 minutes before they expire

- If a domain resolves to more than one address, it automatically round-robins between them to distribute load

- Serve Prometheus-style metrics on 9153/TCP, and provide readiness and liveness checks for Kubernetes

- Load the /etc/hosts.blacklist hosts file (which has just short of 1M domains resolved to 0.0.0.0), reloads it every hour, and does not provide reverse lookups for performance reasons

- Listens on 53/UDP for regular plain-text DNS questions (LAN only), and on 853/TCP for DNS-over-TLS questions, which I have NAT'd so that I can use it when I'm outside

The domain blacklist I generate nightly with a Kubernetes CronJob that runs a Bash script[9]. It essentially pulls and deduplicates the domains in the "safe to use" domain blacklists compiled by https://firebog.net/, as well as removing (whitelisting) a couple hosts at the end.

That's pretty much it. The only downside to this set up is that CoreDNS takes just short of 400MiB of memory (I guess it keeps the resolve table on memory, but 400MiB!?) and lately I'm seeing some OOM restarts by Kubernetes, as it surpasses the 500MiB hard memory limit I have on it. A possible solution might be to keep the resolve table on Redis, which might take up less memory space, but I'm still to try that out.

[1] Which I find MUCH superior to DNS-over-HTTPS. The latter is simply a L7 hack to speed up adoption, but the correct technical solution is DoT, and operating systems should already support it by now (AFAIK, the only OS that supports DoT natively is Android 9+).

[2] It was when I discovered CoreDNS' cache prefetching that I convinced myself to switch to CoreDNS.

[3] http://www.thekelleys.org.uk/dnsmasq/doc.html

[4] It gives you very few stats. I also had to write my own Prometheus expoter[5] because Google's[6] had a fatal flaw and no one answered to the issue. In fact, they closed the Issues tab on GitHub a couple months after my request, so fuck you, Google!

[5] https://github.com/ricardbejarano/dnsmasq_exporter

[6] https://github.com/google/dnsmasq_exporter (as you can see the Issues tab is no longer present)

[7] https://github.com/ricardbejarano/coredns, less bloat than the official image, runs as non-root user, auditable build pipeline, compiled from source during build time. These are all nice to have and to comply with my non-root PodSecurityPolicy. I also like to run my own images just so that I know what's under the hood.

[8]

  local:65535 {
    ready
    health
  }

  (global) {
    log
    errors

    cache 86400 {
      prefetch 5 10m 10%
    }
    dnssec
    loadbalance

    prometheus :9153
  }

  (cloudflare) {
    forward . tls://1.1.1.1 tls://1.0.0.1 {
      tls_servername cloudflare-dns.com
    }
  }

  (blacklist) {
    hosts /etc/hosts.blacklist {
      reload 3600s
      no_reverse
      fallthrough
    }
  }

  .:53 {
    import global
    import blacklist
    import cloudflare
  }

  tls://.:853 {
    import global
    import blacklist
    import cloudflare
    tls /etc/tls/fullchain.pem /etc/tls/privkey.pem
  }
[9]

  #!/bin/bash

  HOSTS_FILE="/tmp/hosts.blacklist"
  HOSTS_FILES="$HOSTS_FILE.d"

  mkdir -p "$HOSTS_FILES"
  download() {
    echo "download($1)"
    curl \
      --location --max-redirs 3 \
      --max-time 20 --retry 3 --retry-delay 0 --retry-max-time 60 \
      "$1" > "$(mktemp "$HOSTS_FILES"/XXXXXX)"
  }

  # https://firebog.net/
  ## suspicious domains
  download "https://hosts-file.net/grm.txt"
  download "https://reddestdream.github.io/Projects/MinimalHosts/etc/MinimalHostsBlocker/minimalhosts"
  download "https://raw.githubusercontent.com/StevenBlack/hosts/master/data/KADhosts/hosts"
  download "https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.Spam/hosts"
  download "https://v.firebog.net/hosts/static/w3kbl.txt"
  ## advertising domains
  download "https://adaway.org/hosts.txt"
  download "https://v.firebog.net/hosts/AdguardDNS.txt"
  download "https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt"
  download "https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt"
  download "https://hosts-file.net/ad_servers.txt"
  download "https://v.firebog.net/hosts/Easylist.txt"
  download "https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts;showintro=0"
  download "https://raw.githubusercontent.com/StevenBlack/hosts/master/data/UncheckyAds/hosts"
  download "https://www.squidblacklist.org/downloads/dg-ads.acl"
  ## tracking & telemetry domains
  download "https://v.firebog.net/hosts/Easyprivacy.txt"
  download "https://v.firebog.net/hosts/Prigent-Ads.txt"
  download "https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt"
  download "https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.2o7Net/hosts"
  download "https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt"
  ## malicious domains
  download "https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt"
  download "https://mirror1.malwaredomains.com/files/justdomains"
  download "https://hosts-file.net/exp.txt"
  download "https://hosts-file.net/emd.txt"
  download "https://hosts-file.net/psh.txt"
  download "https://mirror.cedia.org.ec/malwaredomains/immortal_domains.txt"
  download "https://www.malwaredomainlist.com/hostslist/hosts.txt"
  download "https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt"
  download "https://v.firebog.net/hosts/Prigent-Malware.txt"
  download "https://v.firebog.net/hosts/Prigent-Phishing.txt"
  download "https://phishing.army/download/phishing_army_blocklist_extended.txt"
  download "https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt"
  download "https://ransomwaretracker.abuse.ch/downloads/RW_DOMBL.txt"
  download "https://ransomwaretracker.abuse.ch/downloads/CW_C2_DOMBL.txt"
  download "https://ransomwaretracker.abuse.ch/downloads/LY_C2_DOMBL.txt"
  download "https://ransomwaretracker.abuse.ch/downloads/TC_C2_DOMBL.txt"
  download "https://ransomwaretracker.abuse.ch/downloads/TL_C2_DOMBL.txt"
  download "https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist"
  download "https://v.firebog.net/hosts/Shalla-mal.txt"
  download "https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.Risk/hosts"
  download "https://www.squidblacklist.org/downloads/dg-malicious.acl"

  cat "$HOSTS_FILES"/* | \
  sed \
    -e 's/0.0.0.0//g' \
    -e 's/127.0.0.1//g' \
    -e '/255.255.255.255/d' \
    -e '/::/d' \
    -e '/#/d' \
    -e 's/ //g' \
    -e 's/  //g' \
    -e '/^$/d' \
    -e 's/^/0.0.0.0 /g' | \
  awk '!a[$0]++' | \
  sed \
    -e '/gamovideo.com/d' \
    -e '/openload.co/d' > "$HOSTS_FILE"

  rm -rf "$HOSTS_FILES"
Re:Ask HN: What do you self-host? by tptacek on Sunday October 13, 2019 @12:13PM (#21241145)
DoH isn't an "L7 hack to speed up adoption". It's a DNS privacy mechanism that can't easily be disabled by network administrators, unlike DoT. You may have lots of good reasons to want to disable DNS privacy on your own network, and by all means use DoT to do that. But DoH is superior for end-users.
Re:Ask HN: What do you self-host? by ricardbejarano on Sunday October 13, 2019 @12:24PM (#21241223)

  s/mechanism/hack/g
DoH is a hack, the use of suboptimal protocol to achieve the same goal.

DoT is a protocol explicitly designed for it's purpose.

If brickhead sysadmins block DoT it's their problem, and if you have to work around that then it is, in fact, a hack (or a "workaround", doesn't matter).

It's not that DoT or DoH are superior to one another, it's that DoT is "DNS in TLS", and DoH is "DNS in HTTP in TLS", doesn't that raise a red flag for you?

Re:Ask HN: What do you self-host? by tptacek on Sunday October 13, 2019 @12:29PM (#21241243)
You don't seem to follow. Millions of end-users get access to the Internet through major ISPs that monitor, log, monetize, and manipulate DNS. I'm on AT&T, and they absolutely do this. The purpose of DoH is to add a privacy mechanism that AT&T, or coffee shop wireless networks, or airplane wireless, or whatever, can't trivially disable. That's why it exists and why it's tunneled through HTTPS. Meanwhile, the reason network operators have a meme now about how much better DoT is comes down to the fact that they have middleboxes on their networks that passively monitor DNS, and they themselves (and, more importantly, the vendors that sell those boxes) want to hold back DNS privacy --- at least on their networks --- to keep those boxes working. They prefer a DNS privacy mechanism that has a kill switch that the network controls, not the user.

The idea that end-users should give a shit about any of this "L7" "purpose built" "control plane" "layering violation" nonsense, and opt themselves into a version of DNS privacy that their network operators can turn off for them without end-user consent, is lunacy; bamboozlement.

Re:Ask HN: What do you self-host? by ricardbejarano on Sunday October 13, 2019 @01:32PM (#21241576)
Nothing keeps an end-user from rescinding it's ISP contract as soon as they ever slightly cross the line of filtering a single packet.

I agree in that end-users shouldn't give a dime about DNS privacy, it should be private by default, but it is up to us to promote the correct protocol over the "hacky" one.

If DNS-over-HTTPS is superior, then why don't we shove everything down 443/TCP? Or better yet, why don't we get rid of TCP altogether and send everything over a port-less encrypted dynamically-reliable trasport protocol? Surely middleman couldn't distinguish between traffic.

Ports are there for a reason. The fact that they are used with anti-end-user intent doesn't make them (or any protocol that runs on them) inherently bad. Yet one thing that makes a protocol better than another one, given set of requirements, is efficiency.

By the way, if I were to switch my DoT server from 853/TCP to 443/TCP, the port wouldn't be a problem anymore. Per your standards, now DoT would be better than DoH, wouldn't it? Same results, smaller payloads.

Re:Ask HN: What do you self-host? by tptacek on Sunday October 13, 2019 @01:38PM (#21241606)
Why would any end-user care about what the "correct" protocol was, when the choice was between a privacy protocol with an ISP-owned kill switch and one without?
Re:Ask HN: What do you self-host? by ricardbejarano on Sunday October 13, 2019 @02:14PM (#21241838)
You haven't answered my question. Your supposed "kill switch" wouldn't exist in that scenario.

I gave you an apples to apples protocol comparison. If you tell me there's a single bit that lets you distinguish between HTTPS traffic and DoT traffic running both on 443/TCP, then I'll buy your "kill switch" argument.

And even if you do, nothing keeps me from saying farewell to my ISP as soon as they press that switch.

Re:Ask HN: What do you self-host? by juliusmusseau on Sunday October 20, 2019 @04:55AM (#21302975)
My ssh server runs on port 443. That way I can connect to it when I'm on the Tim Hortons WIFI.
Re:Ask HN: What do you self-host? by atomi on Sunday October 13, 2019 @11:07AM (#21240750)
I'm using Unbound running on a dd-wrt router for a lot of this same functionality. There are some things I don't do only because they aren't a priority for me. But I certainly got DNS over TLS and DNS blocking going.
Re:Ask HN: What do you self-host? by ricardbejarano on Sunday October 13, 2019 @12:06PM (#21241117)
Didn't know unbound had DoT.

I learnt about CoreDNS because Kubernetes uses it for service discovery, and once I read about it's "chaining plugins" philosophy I wanted to try it out.

And it was so refreshing coming from Dnsmasq that I fell in love with it.

Re:Ask HN: What do you self-host? by whycombagator on Sunday October 13, 2019 @01:51PM (#21241690)
Thank you for going into such depth! This is really helpful
Re:Ask HN: What do you self-host? by ricardbejarano on Sunday October 13, 2019 @02:14PM (#21241841)
Happy to help!