However, Consul builds tons more on top of the servers. Consul has a first class notion of services and health checks. It has agents that run on every node to register services, perform health checks, and provide service discovery to local apps via HTTP or DNS.
For every Consul feature there is probably a similar library or tool to do the same thing with Zookeeper. Consul just chose to focus on the service discovery problem and address it as a first class feature of the project.
Full disclosure: I work for HashiCorp, but not on Consul.
Consider that the Consul binary distribution is a zip file containing a single file. Compare that to Zookeeper's tarball which contains 1538 files. That maybe isn't a fair point on which to make a comparison, but I think it's an important signal about their respective approaches. Various people might have different reactions to those numbers, so I'm not trying to pass judgment here. Just giving an idea of one way to think about differences.
Installing a jvm?
I'm big fan of Zookeeper, and agree with parent that is not only industry standard, but it is very robust and battle tested.
Having said that, this is not what Zookeeper was meant to. ZK is coordination service, if you have a distributed service that you need to coordinate and you don't want any single point of failure ZK is the choice.
ZK can be used for service discovery, but that's not the proper use, you could equally set up a single database node, or MongoDB cluster to accomplish essentially the same thing.
So specialized service is better, but I also don't like how Consul solves service discovery. If you think about it, SD doesn't require strong consistency, it is not a big deal if all nodes won't learn about each other at the same time, arguably that's actually preferred.
By removing strong consistency (the only reason why ZK is even compared to Consul) we could have a better SD that's also more scalable.
Can you go into more detail about this? Each computer has its own Consul agent that monitors the health of services running on it. You also get configuration from the agent on the individual computer. All of the agents communicate peer to peer and with the server don’t they? Isn’t that eventually consistent? I’m just starting to use Consul for configuration and haven’t started working on the service discovery part yet.
Because of this design, the don't get benefits of any of them.
Side note, KV is actually strongly consistent, which is what is desired, the thing is that since KV comes with consul it encourages people to use it.
One just need to keep in mind, that Raft and Paxos expect all nodes to be updated on every change (e.g. if you have 5 server node cluster all 5 nodes need to "sign on it" before it is accepted). This is counter-intuitive since adding nodes to cluster makes it actually slower (the goal is correctness and resilience not performance). Basically you just need to keep in mind to not use KV for some write heavy operations. Serving configuration files generally is fine.
Also, key value store reads are usually done locally, from the agent, which goes over gossip and is eventually consistent.
Health checks are also eventually consistent. They are done by the local agent, and through gossip are sent to the masters group.
> Also, key value store reads are usually done locally, from the agent, which goes over gossip and is eventually consistent.
That's not what the documentation says. The KV is using raft, you can relax some guarantees for reading by having clients use consistency=stale, but you're still communicating with the server, except in stale you only communicate with one instead all of them.
> Health checks are also eventually consistent. They are done by the local agent, and through gossip are sent to the masters group.
Also that's not what the documentation says. Gossip is used for adding/removing nodes and the overall health of the node (whether host is reachable) is also done through gossip, but the actual health checks (whether a service is running) is done through consensus protocol.
I still want to emphasize that using consensus protocol for SD is not a good idea. Let say you're running in AWS and follow the best practices which is spreading your servers over multiple AZ. Let say you use us-east-1 which has 6 AZ.
An AWS starts having network issues (like few months ago) and 2 AZs have network issues and can't communicate with the rest. The machines are still running, just can't communicate. It just happens that on 2 of those AZs you run 2 out of your 3 consul servers. Even though all the machines are healthy and running your Consul cluster is down, even for the remaining 4 AZs that don't have any problems.
Consul uses one raft cluster per "datacenter", so you don't have to worry about write performance seriously degrading as you scale, it will only be up to the max of the number of availability zones you'd like to operate in per region. Plus you shouldn't be doing write heavy KV operations with consul anyway.
Just because you are using strong consistency doesn't mean you can't get reasonably high availability in practice.
How do you come to the conclusion that you're communicating with the server? When I'm doing any type of communications with consul via the API I'm always communicating with 127.0.0.1/....
I arrived at that personal assertion by building a SD setup using our conventions + Curator + ZK. In disaster scenarios we forced into the system when testing, I was able to get ZK to basically lock up because, really, that's what it's supposed to do, guarantee consistency--if it can't guarantee consistency, then it tells you about it. Consul for us fit the bill better.
Note: This was some time ago and I do not claim to be a ZK expert; it's quite possible the root issue was me. YMMV, etc etc.
So I welcome a competition in this space.
From my experience ZK is one of the tools that's most ops friendly (i.e. you set it and forget it), right next to Postgres and Solr.
Never used Solr, but even as someone who loves Pg over every other RDBMS out there I'd hardly call it "ops friendly", with the Bring-Your-Own-Tooling mentality it turns a lot of operations people off when they realize features like replication, log archiving, etc. require bolt-on tools to actually do anything useful.
ZK is set and forget though, outside of occasional tweaking of JVM parameters based on growth.
We currently have 9.6 with barman for streaming backups and repmgr to control replication. Compared to any of our MySQL setups is a breeze to manage, my team is owning both and the bad thing is that there was only one issue so far[1] on call (bad, because if system doesn't break often no one will learn about it).
PostgreSQL + barman + repmgr is much nicer than anything else we have for MySQL.
Few things: barman offers streaming backup, meaning it is not just once a day, but it is done continuously. On top of that contains a single command that can verify it operates correctly (it has enough backups, streaming operation works etc).
I also like that everything is robust. In MySQL if replication breaks, I need to manually restart it, if it gets too far behind (too many changes, or it is not fixed fast enough) the wal logs will be erased and I would need to set it up from the scratch again[2].
Postgres is smart enough to keep track of all wal logs and only deletes them if all standby nodes received them. If connection is broken (for example a network partition) it will reconnect and resume replication. It self heals whenever it cans. It is a breeze to manage.
Another thing MySQL lost a lot of points for me was at my previous workplace. An MySQL host suddenly started throwing errors to clients. Turns out that few tables got corrupted, the fixing was easy, just shutting down and running repair tool. What was puzzling is that there was no obvious reason (the disk had plenty of free space, no corruption) the machine was not abruptly terminates (~100 days uptime) I never found why data got corrupted. Also, I never experienced things like that with Postgres.
[1] the issue was that barman host had a bad disk, so that triggered an alert and after disk replacement things continued to work.
[2] there is also one crazy system that MySQL allowed to happen. Essentially in master-standby replication someone had genius idea to do some processing on the data in master and write it in standby. Over time that data became also important, so actually the standby node needed to be backed up instead of the master. This is "great" because if replication breaks, whomever is on call will have fun time doing some acrobatics to restore the whole thing. Postgres correctly doesn't allow writes to standby during replication.
I use them all, as someone who is very familiar with PostgreSQL and Linux systems in general I have no complaints - but unless you know what you're doing it's painful to get all of these set up, they're not intuitive and not integrated in.
This is what I mean by "Bring-Your-Own-Tooling", the core support for replication, wal archiving, etc. is all built-in to PostgreSQL but there's no included tooling to manage it. It's up to you to pick between barman, wal-e, whatever to handle backups. It's up to you to choose between repmgr or pglookout. It's up to you to choose between pgBouncer or pgpool. Batteries not included, and nothing is opinionated, have fun.
I'm not comparing PostgreSQL against MySQL either, I'm comparing it against Oracle or SQL Server - this functionality is baked in and as a result DBA's are a dime a dozen (for the most part), because their job mostly revolves around managing performance and general maintenance instead of wiring a bunch of extra tools together to get it into a production-ready state.
I love PostgreSQL as a developer, and as someone who knows what he's doing I don't mind it from an operations role either - but I don't look forward to the day when we finally fill our open DevOps position and I have to train them on monitoring and managing it.
MySQL replicas self-recover via MASTER_CONNECT_RETRY and MASTER_RETRY_COUNT. e.g. set to 1, 8640000 respectively to make a replica retry every one second for up to 100 days before finally giving up.
At least this is true for network errors. If replication breaks due to data inconsistency, it should stay broken and a human should look into it, IMHO.
Corrupted tables and repairs suspiciously sound like MyISAM, and a realize this ancient engine still lurks at some places. I haven't done a "repair" for over 14 years -- the last time I used MyISAM in production. Using MyISAM today is analogous to using the ext2 file system today.
In any case this just clarifies a couple comments on MySQL and has nothing to say on MySQL vs. PostgreSQL.
And the Consul API is rather nice.
I'm not asking this passive aggressively, I mostly want to know what features it adds over a vanilla Zookeeper setup.