News for hackers. Stuff that matters.
Re:HashiCorp Consul 1.2: Service Mesh by syllogism on Tuesday June 26, 2018 @05:19AM (#17399024)
If you're using Consul for web services, I really recommend the Traefik web server: https://traefik.io

Traefik replaces Nginx: it's the reverse proxy that maps the incoming requests to your various services, which are advertising on some arbitrary localhost port.

The amazing thing is that Traefik integrates with Consul: you only need to point it to your Consul endpoint, and it can automatically publish your services! You can also do other dynamic configuration of Traefik, e.g. by publishing a service via a REST API, via Kubernetes, etc.

I've struggled for years to get Nginx configured correctly, and it's been frustrating to have no alternative. In Nginx, dynamic binding is a premium feature. In the free version, you have to rewrite the config and restart the service. That's not fun if you expect services to come and go as part of your natural life-cycle.

Traefik's still pretty young. Notably, the docs are shit. But the application is well-designed, and it compiles as a single self-contained binary (as it's written in Go).

Re:HashiCorp Consul 1.2: Service Mesh by hellcow on Tuesday June 26, 2018 @09:39AM (#17400457)
Traefik isn't worth the trouble. We used it in production over the course of 2 years, and it was consistently our only source of downtime due to rediculous bugs: not closing file descriptors, breaking changes, and silent failures with no log output, panic or exit even with debug logs enabled.

As you mentioned, the docs are terrible. What makes that worse are the undocumented breaking changes between each release. They don't even pretend to follow semver, so v1.5 broke v1.4, and v1.6 broke v1.5. Each update you pray that it doesn't take your whole setup down. If anything goes wrong, since nothing is documented and there's often no logs explaining what went wrong, you might be down for an extended period while you make 100 best-guess changes to the config that worked in staging, but for whatever reason isn't working in production. May the odds be ever in your favor.

Last I checked, Traefik was 988,000 (!!!) lines of code. That's 20x the size of my very complex web application. I replaced it with 500 lines of go providing all the essential features for me. Higher reliability, way fewer bugs, no breaking changes.

Re:HashiCorp Consul 1.2: Service Mesh by emilevauge on Tuesday June 26, 2018 @05:43PM (#17404232)
Traefik creator here. Wow, that's harsh!

You may encountered issues while using Traefik so giving your opinion is totally fine, but I don't think that's fair to overreact.

Many users (and I mean big companies) have been using Traefik in production for years without issue. I'm not saying there is not bug, which software can claim this, I'm just saying that many users have a good opinion on Traefik stability.

We follow semver, there shouldn't be any breaking change between 2 minor versions. But, yes, it can happen, sometimes, we may have forgot to check a specific use case. But hey, again, let's be fair, we don't want it. We are just human. And no, this does not happen at every minor version and this is pretty uncommon...

Finally, on Traefik size. You are including Traefik dependencies, in vendor/, which is a bit weird. In go, the convention is to push the dependencies in your repository to get reproducible builds, so that's not a good way to count. If you exclude vendor/:

golocc --no-vendor ./...

Lines of Code: 58532 (2987 CLOC, 55545 NCLOC)

Which is rather tiny.

So all in all, I regret you had such a bad experience with Traefik, but I just wanted to express the fact that many users are using it without any issue :) I would be happy to discuss further on this.

Re:HashiCorp Consul 1.2: Service Mesh by shaklee3 on Monday July 02, 2018 @08:56AM (#17441232)
Thanks for clearing all that up.
Re:HashiCorp Consul 1.2: Service Mesh by seeekr on Tuesday June 26, 2018 @10:08AM (#17400681)
Re LOC: Found the 988k LOC number hard to believe, so I checked. If you count the vendored dependencies, then it's indeed a lot, about 1M LOC. Traefik itself though is comparatively a lot less at about 60k lines of Go code with another 8k or so of scripts, config etc.
Re:HashiCorp Consul 1.2: Service Mesh by 1_player on Tuesday June 26, 2018 @11:10AM (#17401142)
Any suggestion? I'm stuck on this problem right now at work, don't like Traefik at all for the same reason as you, liked Docker Flow Proxy but it's not configurable enough and, similarly to Traefik, the documentation is quite bad if you need to do something a little more complicated (like, IP access control!).

Fabio might be a good alternative, but we don't need Consul right now, so I don't want to have to manage yet another puzzle piece.

Should I go with nginx?

Re:HashiCorp Consul 1.2: Service Mesh by borplk on Tuesday June 26, 2018 @11:44AM (#17401501)
Checkout envoyproxy.io
Re:HashiCorp Consul 1.2: Service Mesh by redmar on Tuesday June 26, 2018 @01:49PM (#17402649)
or https://github.com/datawire/ambassador (which wraps envoy with a couple of extra (usability?) features)
Re:HashiCorp Consul 1.2: Service Mesh by syllogism on Tuesday June 26, 2018 @10:09AM (#17400692)
Thanks for the warnings! We don't have it bearing much load yet, so we haven't encountered those things.
Re:HashiCorp Consul 1.2: Service Mesh by vidarh on Tuesday June 26, 2018 @09:52AM (#17400563)
With nginx, you can do dynamic binding in the free version in at least two ways:

1. If you "just" want to map a hostname, to a private IP, you can assign the hostname to a variable, and use the variable in your backend config. This works because Nginx resolves static addresses at start, but resolves names pulled from variables at runtime.

2. If you need to map ports as well, you can use a Lua or Mruby script. E.g. I have a blog post on doing it with mruby here [1], and it's run production sites for a couple of years. This options lets you integrate against pretty much whatever you want.

It's extra hassle, though, of course.

[1] http://hokstadconsulting.com/nginx/mruby-virtualhosts

Re:HashiCorp Consul 1.2: Service Mesh by jorge-fundido on Tuesday June 26, 2018 @10:20AM (#17400768)
For those interested in a lua example, from this article [1], I found & cherry-picked bits from here [2]. The key ingredients (to aid searching) are init_worker_by_lua_block & balancer_by_lua_block.

[1] https://medium.com/@sigil66/dynamic-nginx-upstreams-from-con...

[2] https://github.com/sigil66/nginx-consul-cookbook

Re:HashiCorp Consul 1.2: Service Mesh by matthewmacleod on Tuesday June 26, 2018 @05:40AM (#17399127)
In the free version, you have to rewrite the config and restart the service. That's not fun if you expect services to come and go as part of your natural life-cycle

If that's happening super-frequently I could see the problem, but in practice I'd expect you to use `consul-template` and issue a `reload` to nginx to make it reload configurations with no downtime. This is the solution I've used and it works pretty well.

Re:HashiCorp Consul 1.2: Service Mesh by poooogles on Tuesday June 26, 2018 @06:34AM (#17399361)
>This is the solution I've used and it works pretty well.

Just make sure consul-template never writes bad config as that will stop everything reloading. That's bit me a couple of times.

Re:HashiCorp Consul 1.2: Service Mesh by dewey on Tuesday June 26, 2018 @08:20AM (#17399921)
Can't you just run a nginx -t to run the config test and see if it passes before you reload?
Re:HashiCorp Consul 1.2: Service Mesh by poooogles on Tuesday June 26, 2018 @11:57AM (#17401637)
nginx does that already. The problem is it'll stop you being able to reload configuration on other bits of config. Once there's a config error nothing can reload.

Limited blast radius don't really apply when it comes to nginx config errors (which given its origins is understandable). One error in one bit of config and it all stops.

Re:HashiCorp Consul 1.2: Service Mesh by tacticus on Tuesday June 26, 2018 @06:03AM (#17399238)
Old workers stay around until all the connections for them or their grace period has expired. So i hope you're not too close to the resource limit to handle many generations of workers sticking around. also if you have many services behind a single port with path based routing and do a deploy the connection will stick around on the old workers until everything dies.

It's super fun with haproxy :(

Re:HashiCorp Consul 1.2: Service Mesh by oblio on Tuesday June 26, 2018 @06:43AM (#17399400)
Kind of bad, but what's the alternative? Cut everyone off on reload?
Re:HashiCorp Consul 1.2: Service Mesh by tacticus on Tuesday June 26, 2018 @07:46AM (#17399714)
Have a better proxy that doesn't need to create a new generation of workers for config changes.

Like traefik or envoy or nginx when you pay for it or use the lua shit to do backend discovery (haproxy also has lua shit iirc)

Re:HashiCorp Consul 1.2: Service Mesh by pas on Tuesday June 26, 2018 @10:02AM (#17400646)
Could you elaborate on what's wrong with using lua with ningx, especially as used by kong?
Re:HashiCorp Consul 1.2: Service Mesh by timdumol on Tuesday June 26, 2018 @12:11PM (#17401802)
I think GP means Lua "shit" in a nonderogative manner (synonymous to Lua stuff).
Re:HashiCorp Consul 1.2: Service Mesh by pas on Tuesday June 26, 2018 @10:01AM (#17400635)
kong. uses lua. has a wonderful API.
Re:HashiCorp Consul 1.2: Service Mesh by swills on Tuesday June 26, 2018 @09:48AM (#17400529)
You can also use the consul k-v store as your source of traefik config and then update the config in consul and it updates in traefik instantly without any restart.
Re:HashiCorp Consul 1.2: Service Mesh by cremp on Tuesday June 26, 2018 @08:34AM (#17400019)
We tried Traefik, but ultimately went with Fabio, which also integrates with Consul nicely; plus it does tcp based stuff for unencrypted gRPC services we shuffle around (which at the time, traefik didn't have.)
Re:HashiCorp Consul 1.2: Service Mesh by garettmd on Tuesday June 26, 2018 @09:24AM (#17400354)
How production-ready is Fabio? It looks interesting but I can't tell how mature of a product it is
Re:HashiCorp Consul 1.2: Service Mesh by cremp on Tuesday June 26, 2018 @09:53AM (#17400576)
It's okay for our use case. Honestly, we were just wanting to get a load balancer up and working with consul, and we had grpc services and internal http apps to balance.

I'm not going to lie; it works and does failover correctly, quickly, and plays nice with consul. Not to mention, the name brings back memories...

If I had my way though, we would use HAProxy, but that's just because there is more options, ways to properly route, and more battle-tested.

Re:HashiCorp Consul 1.2: Service Mesh by sagichmal on Tuesday June 26, 2018 @10:20AM (#17400766)
I strongly advise everyone to avoid Traefik. In addition to all the reasons enumerated by hellcow, the code quality is simply abysmal.
Re:HashiCorp Consul 1.2: Service Mesh by emilevauge on Tuesday June 26, 2018 @05:16PM (#17404072)
Could you elaborate? I would never say that the code is perfect, like any other "big" open source project, but we follow a pretty strict review process on each PR (3 LGTM from maintainers). I'm curious to know why you are so negative.