News for hackers. Stuff that matters.
Re:True Zero-Downtime HAProxy Reloads by ksri on Tuesday April 14, 2015 @01:58AM (#9372350)
Why not simply let DNS handle the routing and load balancing?

Bring up another HAProxy with the new configuration. Then swap the new and old IP addresses in DNS. Wait for DNS to propagate. When traffic on old HAProxy is zero, bring it down.

Re:True Zero-Downtime HAProxy Reloads by mrphoebs on Tuesday April 14, 2015 @02:14AM (#9372403)
DNS is a fragile system to use within a DC to do service/endpoint discovery. That's because DNS tends to be a single point of failure.

Though the DNS system can inherently be resilient, within a DC most people only operate a single DNS server because a hierarchical domain scheme and DNS setup within DC is too cumbersome and is much less reactive to end point changes. Eg: Changing a service endpoint in an emergency takes way too long.

A single DNS server means, still there are propagation delays(local caches) and a single point of failure at critical moments(when there is a thundering herd)

Re:True Zero-Downtime HAProxy Reloads by hueving on Tuesday April 14, 2015 @04:59AM (#9372940)
That's a pretty weak argument. It's like calling a databases a single point of failure because the people who set them up tend to be too lazy to setup HA.
Re:True Zero-Downtime HAProxy Reloads by jolynch on Tuesday April 14, 2015 @02:31PM (#9376311)
Heh, the number of times DNS has bitten me in production... My main objections to DNS are:

1) You have no control over clients, there is absolutely no rule that says clients have to respect TTLs (I'm looking at you Java)

2) We are talking about hundreds of HAProxy load balancers, thousands of clients and hundreds of backend services which are moving around all the time. I just honestly didn't want to deal with DNS propagation limiting my flexibility.

3) At least at Yelp, we don't really have particularly nice control apis for DNS. This is sort of specific to Yelp, but it was a factor.

Re:True Zero-Downtime HAProxy Reloads by meatmanek on Tuesday April 14, 2015 @02:14AM (#9372404)
DNS is surprisingly tricky as a service discovery tool. A lot of clients are poorly behaved, and will cache values for too long (or forever). It'd also be another dependency critical for site functionality.
Re:True Zero-Downtime HAProxy Reloads by nierman on Tuesday April 14, 2015 @02:49AM (#9372501)
Don't muck with the DNS record. Use an "A record" associated with a secondary/floating IP; then simply move the IP associated with the A record to the new machine and issue a gratuitous ARP.

You can avoid setting arbitrarily low--but not low enough--TTL, etc.