News for hackers. Stuff that matters.
Re:HAproxy in the era of Microservices by fidget on Monday October 26, 2015 @11:39AM (#10452120)
I'm moving away from HAProxy. The lack of zero downtime config reloads is consistently causing problems, especially in a high reload environment (i.e. using bamboo [0]). One team deploys an application that goes into a restart loop, changing our routing table constantly? Well there goes our 99th percentile out the window.

That said, would prefer not to be using nginx, so if anyone has any recommendations for high quality zero-downtime-reloadable HTTP proxies, I'd be very interested.

[0]: https://github.com/QubitProducts/bamboo

Re:HAproxy in the era of Microservices by jolynch on Monday October 26, 2015 @11:44AM (#10452150)
Check out the deep dive I wrote into how you can help prevent HAProxy reloads from dropping traffic without a large 99% hit over at http://engineeringblog.yelp.com/2015/04/true-zero-downtime-h...

Looks like bamboo encourages folks to use this strategy via a custom reload command: https://github.com/QubitProducts/bamboo/issues/152

Re:HAproxy in the era of Microservices by fidget on Monday October 26, 2015 @12:02PM (#10452263)
That's egress traffic :) And yeah, https://github.com/QubitProducts/bamboo/issues/143#issuecomm... is currently my best idea (with some additional nfq stuff). Doesn't help when requests span the entirety of the reload though.
Re:HAproxy in the era of Microservices by jolynch on Monday October 26, 2015 @12:21PM (#10452383)
I mean, I wouldn't encourage anyone to use HAProxy for microservice load balancing unless they're running it on every node, at which point clients are always talking to localhost and everything is egress.

That being said, you can turn ingress into egress with an ifb. A coworker of mine created a proof of concept for using our strategy with an external facing load balancer but I don't think he ever tried it in production so I'm not sure how well it works.

To be fair I'm scared of both the iptables and tc solution on external LBs because you might never know that you're refusing connections accidentally. Linux 4.4 is coming with some patches that should help with this and afaik BSDs have done it right since the start.

Re:HAproxy in the era of Microservices by falcolas on Monday October 26, 2015 @02:47PM (#10453506)
Live configuration via the stats socket is frequently overlooked, but works a treat. I'd recommend investigating it before giving up on haproxy.
Re:HAproxy in the era of Microservices by fidget on Tuesday October 27, 2015 @03:40AM (#10456726)
I use it post reload to persist the state of health checks, which I thought was pretty cool. However, I don't believe you can add a (new) server to a backend via the stats socket.
Re:HAproxy in the era of Microservices by falcolas on Tuesday October 27, 2015 @10:59AM (#10458359)
> However, I don't believe you can add a (new) server to a backend via the stats socket.

Yes, though a bit of advanced planning and creative configuration can usually work around this limitation.

Making the assumption that new servers and services occur within a predictable range of IPs and ports (or have their own haproxy to route within the server), you can create a list of backend servers which are all disabled and just waiting to be individually enabled.

I've done this in the past, and the disabled server entries never seem to have a negative impact on HAProxy. It also has the advantage that you can set your HAProxy instances to peer with each other, so the change only has to be made once to propagate to connected instances.

Re:HAproxy in the era of Microservices by magiconair on Monday October 26, 2015 @02:40PM (#10453453)
I'm curious whether https://github.com/eBay/fabio would meet your needs. It picks up the routing table from consul changes and service registrations and reloads it without restart. All the services have to do is to register their routes in consul. It is meant as a replacement for consul-template+HAproxy.

Disclosure: I'm the author

Re:HAproxy in the era of Microservices by fidget on Tuesday October 27, 2015 @03:39AM (#10456724)
Unfortunately consul integration isn't useful for us.
Re:HAproxy in the era of Microservices by TurningCanadian on Monday October 26, 2015 @02:36PM (#10453424)
AFAIK Apache Traffic Server does zero-downtime reload http://trafficserver.readthedocs.org/en/latest/reference/con...