Some applications use SRV records for load balancing. Many VoIP and video conferencing apps do this. There is a better list on Wikipedia.
_service._proto.name. TTL class SRV priority weight port target.
There's actually the germ of an interesting idea in that statement. If I'm going to go to the trouble, let's say, of running a local TCP forwarder (good for the whole device), can I run a packet sniffer at the same time and watch netflows and edit the responses I return to the device based on what I see performance-wise concerning those flows?
Expert me says that web sites are loaded with too much cruft and since the far end terminations are spread far and wide, there's not enough opportunity to apply that learning in any practical sense. But I could be wrong. (https://github.com/m3047/shodohflo)
Typo, meant 853.
Right now, you can try resolving common hosts, and you will see that they often provide several hosts in response to a lookup. What the browser does with those IPs is up to the browser, the standard does not define what to do. What the administrator that sets up that record wants is "send to whichever one of these seems healthy", and some browsers do do that. Other browsers just pick one at random and report failure, so your redundancy makes the system more likely to break.
What I want is a way to define what to do in this case. Maybe you want to try them all in parallel and pick the first to respond (at the TCP connection level). Maybe you want to try them sequentially. Maybe you want to open a connection to all of them and send 1/n requests to each. Right now, there is no way to know what the service intends, so the browser has to guess. And each one guesses differently.
(You will notice that people like Google and Cloudflare skillfully respond with only one record with a 5 minute TTL. That is so the behavior of the browser is well defined, but it also eats their entire year of 99.999% uptime with one bad reply. Your systems had better be very reliable if DNS issues can eat a year's worth of error budget.)
This chapter in the Google SRE book explains how our load balancing DNS works:
https://landing.google.com/sre/sre-book/chapters/load-balanc...
Source: my team runs this service
It might be a little more resilient than even a very minimal nginx, but more than that, I think it must give you more control about what happens when a packet is not "answered" after some set amount of time - you write off who should have been the answerer, then resend that same packet to another server. Keep a buffer of packet, scrape them from the buffer when ACK'ed by the answerer, resend them to another answerer if not ACK'ed after some set amount of time.
Am I guessing correctly?
It seems a bit overcomplicated for normal usecases, but adequate for a large scale like google.
The main thing I wanted to respond to in this thread about a single bad server destroying your yearly SLO is described in the first paragraph in the section on load balancing at the virtual IP address.
Overall, virtual IPs are still an interesting solution.
In the rrDNS, remove the A record of the hosts that fails tests, or that has a load that's too high
> Maybe you want to try them all in parallel and pick the first to respond (at the TCP connection level).
Something a geoIP at your DNS can do, certainly not as good as doing that in the client, but it should be decent enough.
> Your systems had better be very reliable if DNS issues can eat a year's worth of error budget
Or, if you aren't google or cloudflare, use a 30 to 60s TTL in rrDNS, with health checks to selectively remove IP that fail, on pools splitting your servers by region with geoIP - this way, if 1/10 of your east coast servers fail, nobody from APAC will be impacted, and only 1/10th of your US east users, and only for the TTL (I'm abstracting ISP that cache for too long, but you already mitigate a lot of the problem there)
I can see how it would be easier to handle that in the browser, but you may already be able to do that with some JS to estimate the latency, then store the result in a cookie that causes a reload to www.eastcoast.yoursite.com if the user sticks to www.yoursite.com or after returning home goes to www.apac.yoursite.com while new measurement say "not optimal" and update the cookie
I don't really like it because it still means a minute of downtime when the topology does change. I would prefer telling the browser what strategy to use to try a new node, rather than relying on heuristics and defaults.
Nameserver records (NS records) used to locate a resource are served by other nameservers. NS records are chosen from among those offered in response to a query (RRs), and should all be tried if necessary to elicit a response. The algorithm isn't strictly specified and some nameservers will shuffle the order in which they return RRs in their answers, some won't assuming the stub resolver or app will do it. The foregoing also applies to A and AAAA records (returning IP addresses for names), and this has long been used as a quick and easy form of load balancing/failover, except that it doesn't really failover very well unless your app is coded to try all of the different answers (and the stub resolver returns them to your app).
Nameservers querying other nameservers (caching/recursive resolvers) are supposed to compile metrics on response times when they make upstream requests and pick the fastest upstreams once they learn them.
Stub resolvers (running on your device) typically query nameservers in the order you specified them in your network config, but not always.
From the foregoing, you can probably see that running a caching/recursive resolver close to your devices is supposed to be desirable, by design.
So far, so far. ;-)
As specified, and it's never been changed, DNS tries UDP first. "Ok", you think "that must mean it will try TCP" but that's not actually true: it only tries TCP if it receives a UDP response with TC=1 (flagged as truncated). But if there's a UDP frag and it doesn't get all the frags or never gets a UDP response at all it /never/ tries TCP.
You're mixing two very different environments above: 1) a datacenter with (let's just assume) VPCs and 2) a web browser.
In case #2 I'll match your ante and raise you an overloaded segment which is dropping UDP packets, in which case stuff may fail to resolve at all. Oh look, I drew a wildcard: traditionally browsers have utilized the devices stub resolver, but since they've pushed ahead with DoH they've had to implement their own. People think I'm a DNS expert (what do they know?) and I guess conventional wisdom amongst myself and my peers is that UDP should perform better than TCP but anecdotally people are claiming that DoH and DoT perform better for them than their stub resolver. "Must be your ISP messing with you" says someone, "yeah right, that's gotta be it". Me: "did you try running your own local resolver?" them: "wuut?"
So here's where I confess that the experts aren't always right, because I run my own local resolver and I have the same problem: when the streaming media devices are running DNS resolution on the wifi-connected laptop sucks and if I run a TCP forwarder it starts working! (https://github.com/m3047/tcp_only_forwarder).
Now to case #1, the datacenter. I hope you're running your own authoritative and caching server, and you should read about views in your server config guide; using EDNS to pass subnet info is a kludge. If you're writing datacenter apps, you should consider doing your own resolution and using TCP (try the forwarder, I dare you), and provisioning accordingly (because DNS servers assume most requests will come in via UDP).
If you want load balancing "you know, like nginx" I've got news for you: BIND comes with instructions for configuring nginx as a reverse TCP proxy. Oh! Looks like I've got a straight in a single suit: nginx provides SSL termination so I've got DoT for free!
For example, consider the case where you resolve ycombinator.com. You get:
ycombinator.com. 59 IN A 13.225.214.21
ycombinator.com. 59 IN A 13.225.214.51
ycombinator.com. 59 IN A 13.225.214.81
ycombinator.com. 59 IN A 13.225.214.73
Don't get me wrong, I think stub resolver logic is stuck in the 1980s!
If your app or device doesn't have such a history, and no way to obtain it, then maybe the server can do it based on what it knows about its history with IP addresses "close" to yours (the EDNS kludge).
> It doesn't respond. What do I do now? Tell the user ycombinator.com is down? Try another one? All of this could be defined by a standard
I would argue the DNS is clear about this from its own behavior: it tries another one.
Although it's not clear from `pydoc3 socket.create_connection`, it's pretty clear from https://docs.python.org/3/library/socket.html#creating-socke... that socket.create_connection() that it will "...try to connect to all possible addresses in turn until a connection succeeds."
So I would say that the correct action would be to try all possible addresses until one succeeds.
Well, there was an RFC (found it, RFC 3484) that told you to pick the one closest to your network (which wouldn't make a difference in this case, unless you were in say 13.225.214.0/27 or so). But that's not actually helpful, because given two destination IPs, one in the same /8 as me, and one not, I don't have any information that would help me determine which is a better choice.
From experience, most browsers will try a couple IPs before showing an error message, but that's not standard. If you have a fancy authoritative server, a lot of traffic, and a bunch of server IPs, you can get OK balancing by telling some clients some IPs and some clients other IPs; but it depends on having enough diversity in recursive servers; if all of your users are coming from one mobile ISP, chances are you won't get a lot of balancing.
(And I'm sure you already know all this :)
Better to have clients with a bit of intelligence. :)
If you're talking about subprotocols/versions of HTTP like HTTP2 then you can define subservices, so you could have _http2._http. But no one has proposed that yet :)
Of course with anycast, multiple A records can be redundant :)
It is baffling to me that inside the datacenter, I can control the balancing strategy for every service-to-service transaction, but for the end user's browser, all I can do is some L3 hacks to make two routers appear as one (for failover purposes). L3 balancing would be completely unnecessary if I could just program the user agent to go to the right host, after all. The end result is unnecessary cost and complexity multiplied over a billion websites.