This is also what we're trying to do in Tailscale (to grab your MagicDNS domain, and whatever corporate split DNS you set, but not anything else). And yeah, on linux, basically systemd-resolved is the only thing that gets this right (with dynamic config - with fully static config, other recursors are an option, with varying tradeoffs), everything else assumes "one resolver should be enough for everything".

So, your solution to proxy all the traffic and split out as needed is the right way to do it without adding more software into each VM :(

Doing stuff via LD_PRELOAD would be hilarious, you should definitely do that and report back ducks behind the blast shield

It depends on what you're using for the resolver. I'm assuming you only care about gethostbyname(3) and friends. With glibc that means nss; generally you're also looking at libnss_dns.so, which uses glibc's resolv (copied from BIND). This doesn't include enough configuration to do what you suggest; it pretty much just points everything towards a server.

So you have two options: use a different NSS module (maybe write your own?) or have a proxy DNS resolver that sends different requests to different places.

systemd-resolved actually handles the first option pretty well (although it would prefer that you use the dbus interface over gai). It can handle multiple interfaces with separate domains and split DNS fairly well! (Not so good with reverse DNS, unfortunately. But I get it, reverse DNS is pretty hacky anyways.)

If you prefer the forwarder route, dnsmasq seems to be fairly popular these days in the embedded world and elsewhere.

If I were you, I think I'd write a short NSS module or use dnsmasq, depending upon your needs.

Indeed, dnsmasq seems to be the least painful way to provide special DNS zones, special host files, and forwarding for everything else.

Also, your resolv.conf becomes trivial.

The trick is to grab the DHCP-provided DNS server address on reconnections and update the forwarding, if you use a laptop. For VMs in the cloud, it's not a problem, of course.

Couldn't you run dnsmasq, unbound, or some other configurable resolver on the localhost? Though that's probably already what's done with the internal smart resolver.

dnsmasq doesn't do authority checks so unfortunately an external cname to a .internal name may not resolve correctly if the upstream server gives back "additional data" containing the A record answer for the target .internal name. On the other hand, we use unbound instead of dnsmasq to solve this at work and it works great. If you just want to forward internal vs everything else differently it's great though.

If you run CoreDNS instead, they have a kubernetes mode of operation where it gives different answers based on the source IP of the query to optimize search domain lookups. So if the node hosting the VMs here did similar, you could solve this pretty easily using CoreDNS with a custom plugin but I don't love putting all my traffic through CoreDNS vs a more mature DNS server.

bind can do split horizon stuff based on source IP so that could be a potential solution, though I'm not sure if it's capable of split horizon forwarders.

This is proper advice. My recursor of choice is pdns-recursor , but bind in forward-only mode with several forwarders for different zones will work too

You're trying to do something DNS wasn't built for, so, there is no good solution for what you want ("only serve .internal and nothing else"). As I see it, you have 3 options:

1. Use ".internal.someotherdomain.io" and the regular old public DNS. (this probably only does half of what you want)

2. Use dnsmasq at the edge of each .internal subnet, and maybe require the customer to configure your DNS in their container. (I believe this is what AWS does?)

3. Intercept tcp/udp port 53 at the subnet edge, spoof your own responses for .internal, pass through anything else to wherever it's going. Dealing with tcp could be tricky, udp is easy. Probably this will break some clients, and generally be the hardest thing to implement.

I think the best option is #2, as it will actually work reliably and with the least difficulty for everyone.

(2) is what we do, modulo that we don't run dnsmasq. (We slip our DNS configuration in via our init).

I'm not persuaded by appeals to what DNS was or wasn't built for (unfortunately neither is the IETF; to wit: DNSSEC). I agree with you that the ask here is big! But I'm not convinced there isn't a cleaner solution that works within our constraints.

Then another option is to implement all 3. This gives you what you want most of the time, but also supports all customer use cases. The downside is the customer now needs a flow chart to understand how their DNS is resolving.

To start with, first implement (1).

Assuming .internal points to internal-only IPs, and you are reusing internal IPs (each customer VPC can have arbitrary private IPs), you can have ni-172-16-0-5.customer.internal.somedomain.io return A record to "172.16.0.5". (You don't have to add .customer, but you might gain some useful analytics)

A customer needing a hostname for 172.16.0.5 can resolve ni-172-16-0-5.customer.internal.somedomain.io. By default it'll use whatever DNS server/resolver they have configured (or yours with option (2)). The request goes through public DNS until it gets back to somedomain.io (your domain) and you return the result.

Upside: You're not hosting all the customer's DNS resolution. Downside: The customer will see a longer round-trip for .internal requests, and their VPC needs a NAT gateway.

Next implement (3).

Intercept all DNS traffic that matches ".internal.somedomain.io" and spoof responses. Write this tight enough that it will only return responses when you are sure they won't break clients, and pass the rest to the public DNS using option (1), which prevents from having to mess with TCP connections (they'll be slower though the public internet, but it's TCP, what did they expect?).

Upside: You are only returning results for .internal, everything else can use somebody else's DNS, and the customer doesn't need a NAT gateway. Downside: Requires custom engineering which is subtly complex, and is hidden magic the customer doesn't expect, so could cause troubleshooting headaches.

Next, implement (2) as an opt-in.

Have your VPC DHCP server return your internal dns server. You resolve .internal locally and handle the rest as a big hulking resolver. You can also slip this DNS server into their containers, but as a customer I'd want a giant red flag in the docs that advertises this.

Upside: Works in every customer scenario. Bonus: now you can return reverse records. Downside: Now you really are hosting the customer's DNS resolver.

> If there's a clean way to resolve this, so that the VM itself can just send `.internal` queries to us, and everything else to `1.1.1.1` or `8.8.8.8` or whatever the customer's container had, I would _love_ to hear it. I've come pretty close to breaking out preload in anger over this problem.

If you're at that point, rather than preload, why not simply add the code to glibc and contribute it back? Has there been some resistance from the maintainers already?

It would seem like it should be doable to add it on in a safe way, e.g. something that would work for both: `nameserver 192.168.0.1`

and ``` nameserver 192.168.0.1 nameserver 10.10.10.1 .internal .private .etc ```

Even if there's hesitance in accepting a change to the `nameserver` options, you easily make this an option, e.g.: `options tld-nameserver:.internal:<addr>`

This just adds more complexity to a universally used user space library. And also breaks when some programs don't use the same resolver. Just implement your own dns server that does it for you (or reuse dnsmasq).

... I realize I'm a monster, but aren't y'all already extensively using BPF for things? Sticking in a BPF egress filter on the VM that rewrites outbound DNS packets with .internal in the question to point at your DNS server seems like it would be lighter weight than just handling all queries recursively.

The main problem with this is that you need to handle TCP connections, in which the queries do not have to match up to packet boundaries, and the client can multiplex internal and external queries on the same connection.

The interface for DNS from an arbitrary container is the DNS protocol, and since you can’t control the software that is making the queries on the client side of the protocol, you have to do the special logic on the server side.

Do you, realistically? The standard behavior for most things afaik on looking up a domain name is to send out an A query and (simultaneously ideally) A queries with searchdomains appended. That's not gonna get anywhere near 512 bytes for a fly .internal service name, I'd expect.

The requirement is to support arbitrary containers, so it should still work if the container contains stub resolver software that pipelines requests over TCP.

This is good praxis. I have received some thoughts about how to do this without parsing DNS in eBPF (which would be a disaster).

Why do you need kernel-mode parser for that ? iptables -t nat -I POSTROUTING -p udp --dport 53 -j DNAT --to <your internal recursor, sending .internal to authoritative dns server for that zone, and resolving globally available hostnames by itself>

Well, because he doesn't want to be inline for non-.internal DNS queries.

You can technically add an iptables match rule to only forward DNS packets whose contents match a ".internal" DNS query, but it sounds like a recipe for disaster. It would be better if they wrote an actual iptables protocol filter for DNS (assuming one doesn't exist) but that's so much work for so little benefit.

sure, but that's why BPF is so great; I'd guess a reasonable program that assumes port 53, parses the packet enough to look at the first question and compare domain against .internal and then rewrites address + updates checksum is maybe... 300-400 LOC? can probably get a bit fancier, but it shouldn't be too painful to write and will execute plenty fast enough.

> This sucks; we shouldn't have to be inline for arbitrary customer DNS.

Why not? Someone has to do it, and if I were paying Fly for service, I'd rather have Fly handle my DNS queries than be another freeloader on Google or Cloudflare.

This is called "Split Horizon DNS". The problem is, there is no standard way to communicate this information to a machine. e.g., afaik there is no DHCP extension to say "for .internal, please see this server, for the rest, see whatever". So you pretty much have to run a server in the middle.

It -is- possible, as others have commented, to do this within the VM or container but they need a way to get that information. systemd-resolved does this. From what I understand of fly.io, the container is setup by the user, so you're not going to have control over their networking configuration or environment - is that right?

Now there is an alternative, which is instead of using .internal, to use a public facing internet hosting that is delegated to your internal DNS servers. That way they use their normal DNS, and it resolves to your nameserver. But the DNS needs to be public, and critically, if the internet DNS goes down, your .internal resolution will also stop working. Which is highly likely to cause problems for peoples applications. And also adds un-predictable latency to their local service connections which is almost certainly going to cause problems - it's not a good idea.

Having said all of that, generally speaking, it makes sense that -you- provide the DNS. It's pretty much the standard way that your 'ISP', 'Hosting Provider', etc, provides the DNS rather than relying on the public internet DNS. Only a few years ago there was no such public DNS.

The other possible alternative would be to leverage multicast DNS (libnss-mdns/avahi-daemon) or LLMNR but I'm not sure those are really ideal for this use case either. And in both cases requires resolver support that you dont get from the basic glibc resolver. So for your use case, may still not be workable.

So I think you're stuck, unless you can mandate that all of your VM images use systemd-resolved and somehow communicate the config into them.

We do and we don't control the user environment. We can run anything we want in a container, and we can tamper with the runtime environment of a container entrypoint any way we want. But what we come up with has to be compatible with any container someone throws at us.

Additionally: the basic problem of routing DNS requests based on domains is trivial. It's probably less than 100 lines of Rust to send `.internal` to our servers and anything else to a public recursor. But the goal is to get our own code out of the path of people's DNS lookups, even if it's just 100 lines of Rust.

So running dnsmasq or some systemd resolver server thingy doesn't really solve anything for us. Before we did that, we'd just make our `init` a DNS server. But it'd be neat to come up with a way to to solve the problem without having a SPOF server dependency. It might could be possible! You can already see that if you assume a specific libc, you can just preload a libc stub resolver that routes requests. But, of course, we can't assume a particular libc.

Most of my point here is just that "standard" Linux DNS is bad.

Bad compared to what? Is there any OS that has a better DNS resolution process/methodology? Windows? BSD? Solaris? Serious question. I mean do they all suck or?

If you're willing to assume glibc, a reasonably clean approach is to write your own nss_fly module that returns results for .internal.

The downside is that Go code will drop to cgo for all resolutions (because it will see something it can't handle in pure Go in resolv.conf) and non-glibc code (like Alpine containers using musl, or non-libc resolvers like ares) won't get anywhere at all.

But you're kind of doomed in that latter case, anyway, because since there currently isn't a standard for how resolv.conf should express the rules you want, the effort to get it adopted by glibc, musl, Go, Chrome, ares, and all the other various interpreters of resolv.conf will take way too long to make a practical impact on a startup's product.

I can think of a variety of ways. Some involve running some code on the VM. For example, dnscache or dqcache would work. This can also be complished using tinydns. Tiny programs that use very little memory. But it sounds like you are trying to avoid asking the user to install anything other than flyctl on the "bare Linux VM". What is not clear is what programs are installed by default in the "bare Linux VM". The other question is how many ".internal" domains the VM will need to resolve. The simplest solution that comes to mind is when the user provisions an IP, flyctl writes the ".internal" domain(s) for that IP to /etc/hosts. /etc/resolv.conf can then point to whatever the user prefers. IMO, users should be running their own DNS servers, not setting /etc/resolv.conf to point to third party DNS addresses like 1.1.1.1 or 8.8.8.8 or whatever. In the case they are running their own DNS server on the VM, then it becomes trivial segregate internal from external domains. The configuration for dnscache is so easy that it could be done by the flyctl program, requiring no user interaction. (Something like tinydns-config.) One could even have configurations for a variety of DNS servers in flyctl, in case the user prefers unbound, etc.

If it is flyctl doing the resolution then another idea is to have flyctl do the resolution in a chroot with its own resolv.conf.

But shouldn't this be solved with Multicast DNS [1] and DNS based service discovery [2]?

Sounds to me that your .internal TLD is causing a lot of headache and sticking to avahi and .local would be easier?

[1] https://tools.ietf.org/html/rfc6762

[2] http://dns-sd.org

What we deploy has to work with any reasonable application; we build to the lowest common denominator interface.

People on this thread have a lot of ideas about what they'd do if they got to be the sysadmin for every container running on our fleet.

Funnily enough, that is the _exact_ same problem I'm facing right now, down to Firecracker and a custom internal TLD. I'm excited to see the solutions. I think the only difference is that I need to run this DNS service on the same host as my VMs, so I will need to use a different port than systemd-resolve.

I'd recommend to run it on the same port, but different IP (127.0.1.2 , f.e.) , due to inability of some programs to use non-standard port

So apparently systemd-resolve does support redirecting custom TLDs to a specific nameserver [0], as mentioned by dnr. However, support for custom ports was only added in version 246 and my Ubuntu 18.04 ships with 237.

As far as I can tell, my way forward is to use v246, then set my TLD to my custom nameserver with my custom port and add a stub listener on localhost (instead of 127.0.0.53, which is the default). Then I can offer my VMs to use their tap gateway (which is the Ubuntu host) as a DNS server if they want, and that will resolve my custom TLD and forward all other stuff to its own nameserver. Or, if the user wants, they can also do their own stuff.

It seems that a lot has happened with systemd-resolve in the last few months.

systemd-resolved can do this, though I haven't played with it too much yet.

Here's a random post I just found: https://gist.github.com/brasey/fa2277a6d7242cdf4e4b7c720d42b...

(Ha, I did the thing where I read the comments before the post, and the post describes how to do this. So what's still missing?)

I'm not sure how reasonable it is for us to run systemd-resolved (or systemd itself) on our VMs. We're trying to provide a clean environment for any random container to run on; we provide our own init, and that's almost the whole of it.

What, I think, you really care about here is glibc's behavior. But then, you can't depend on any one libc, either.

The cliffs of cloud native software defined networking insanity await you:

https://github.com/containernetworking/cni

https://github.com/firecracker-microvm/firecracker-go-sdk

  Firecracker, by design, only supports Linux tap devices. The SDK provides facilities to:

  Attach a pre-created tap device, optionally with static IP configuration, to the VM. This is referred to as a "static network interface".
  Create a tap device via CNI plugins, which will then be attached to the VM automatically by the SDK. This is referred to as a "CNI-configured network interface"

It's unreasonable to bring the whole cow to just get a gallon of milk . Use standalone resolver.

I used to have a very similar problem (didn’t want my company vpn to handle all dns traffic, only what was directed at the company), I managed to solve it with a local dnsmasq on my machine.

Unlikely to fit your usecase. But it’s not impossible for us plebs.

I guess you've heard of dnsmasq which I'm pretty sure can do all you want.

Perhaps it doesn't count as bare.

[EDIT: not only do many other mention this, but the article does too! my bad]

Yes, we built our own DNS server in Rust, we're aware of dnsmasq. The problem we'd like to resolve is not having to have a SPOF DNS server we're responsible for in between our users VMs and the Internet.

The prospect of running dnsmasq on every VM we launch is to aversive to consider. At that point, we'd just put a DNS recursor in our init.

Multiple redundant servers? I'm way out of my depth here, but if a spof is your main problem, perhaps you just need some redundancy?

How about generating a custom resolv.conf per instance, protecting it from systemd via `chattr +i`, then run any other services you need to inside?

On macOS, you could do this trivially by creating /etc/resolver/internal with the name servers in it.

Linux should copy that.

Systemd-resolved does much more than MacOS resolver. It can make the additional resolvers active conditional on whether the link is up, for example (think VPN). MacOS resolver can't do that.

I thought tptacek was looking for something simpler than systemd-resolved. I was just pointing out the simplicity of /etc/resolver/<domain> on macOS that's been available since, I think, the first version of macOS. It's a nice feature.

In any case, on macOS, you use scutil to adjust resolvers when you change networking state as when bringing up a VPN.

Conceptually they are similar; one has config files in /etc/resolver/*, other in /etc/systemd/resolved.conf.d/*.conf; one has mDNSResponder process (yes, it is also unicast DNS resolver and cache, since 10.6, despite the name) and the other systemd-resolved. With systemd-resolved, you can achieve exactly the same thing as with MacOS resolver, for approximately the same cost.

With systemd-resolved, you don't have to adjust resolvers manually as links come up and down, NetworkManager's VPN connections play nice with it, while DNS resolver handling in Mac VPN software (including Wireguard and Tunnelblick) is a nuasance :(.

Feels like the actual clean way to resolve this is to not use DNS for discovery of named services.