I can't say much but there were multiple issues that could have been easily avoided with a basic understanding of GIS - I have no doubt that some of it was due to the IT department at my employer but am also incredibly surprised that any GIS staff at the ride share companies would have agreed to what was implemented.

Waste of existing talent is unfortunately a way of life at large corporations :( .

5a) ignores MySQL's ability to make many runtime config changes without a restart

6) hard-codes master/slave relationships into the boot process (at least if I'm understanding the config json)

7) adds additional risk (noted as docker crashes) and client issues (noted as "userland proxy" comment)

8) requires additional ops overhead (noted as special care needed for masters)

---

But if you already run everything in docker and really want to get rid of puppet, then why not?

---

this article links to ansible but not puppet and is the only blog post tagged with either one; I think this really boils down to puppet vs $x

Given my limited failed experience with playing with docker + postgres (and various volume stores including flocker) I have continued doubts about Uber's choices.

I suppose they are having success but I wonder at what cost.

Specifically the quote on [1]:

Finally, our decision ultimately came down to operational trust in the system we’d use, as it contains mission-critical trip data. Alternative solutions may be able to run reliably in theory, but whether we would have the operational knowledge to immediately execute their fullest capabilities factored in greatly to our decision to develop our own solution to our Uber use case. This is not only dependent on the technology we use, but also the experience with it that we had on the team.

[1]: https://eng.uber.com/schemaless-part-one/

And you know, I wouldn't argue too much at the thought of running MySQL out of a container. It does avoid a lot of basic issues around upgrades, versioning, stack split, etc. It's the act of treating it like all other containerized software which gets my panties in a twist.

Stateful systems can't be treated like stateless systems if you want to maintain anything resembling reasonable uptime and performance.

Docker does not replace puppet and they clearly wrote that they developed a System just for that.

It depends. @falcolas, could you point out a few other solutions then that? (for resharding live, other then replication / copy / rsync / ...)

The lightest weight solution I've seen is restoring from a daily backup in something like S3, then setting up as a slave from a live master to catch up on the day's binlogs. Still a lot of data to move and load, but at least it's not the entire contents of the DB.

The best you can do is be in control of when data transfers happens so you're doing it when it makes sense and not in the middle of your highest traffic period (which is what frequently happens when attempting to automatically scale DBs in response to load).

Some databases have a configurable limit in MB/s for replication, or it's possible to assign disk/CPU quota on the slave to slow it down.

Combine that with good planning and monitoring, you'll be fine =)

You should always run mysql_upgrade to be safe, and I'd assume Uber's automation does this. But you probably won't encounter problems if you don't.

mysql_upgrade typically just makes some quick changes to the system schema ("mysql" database), not to other data. And it typically only needs to change things across major version upgrades (5.5->5.6) or pre-GA releases; it's rare to make a change in a point release of a major version much after the GA release.

That all said, I don't ever advocate using Docker for MySQL in production at this time; see my longer comment in another subthread below for reasoning.

My experience with this mirrors the parents. The more data you have, and the more obscure engine features you use, the more likely it is to happen. It only takes one major failure to make you want to test the daylights out of any upgrades or downgrades.

The more common classes of problem are performance degradation, config options being added or renamed, new features being buggy in rare edge cases, etc. Not things particular to InnoDB's storage format or things that would prevent a volume from being usable with a different version.

Really? The mysql System Database is filled with tables that hold important configuration information. You can say "add one more command to set up the initial users," but if you have a lot of grants, you've got six levels of database privileges -- global, database, table, host, stored procedure/functions and proxies -- that all need to be configured. You've got the event scheduler. If you have UDFs that you expect on every server, those need to be configured. None of this is in my.cnf. Maybe at the scale you've worked at, you've never had issues with any of those things in a three-command puppet/Chef/Ansible setup, but maybe Uber has problems at a scale that you don't have, rather than them not understanding how MySQL is configured?

Much better to store it as its own version controlled SQL file which is run against a newly started DB (that third command I referenced). You could break out every statement into its own provisioner command, but then you are beginning to muddle the line between machine-level configuration and database level configuration.

So yes, I still stand by my original assertion that you can stand up a fully functional DB with three Ansible/puppet/chef commands.

> maybe Uber has problems at a scale that you don't have

I will admit that it is entirely possible. But from what they present to the public (which is all any of us outside of Uber have to go on), their problems aren't really that unique: a heavy write workload which is easily sharded. Only their chosen method of resolving them is proving to be unique, and they appear to disregard any and all normal methods of resolving their issues.

Sometimes the boring way of doing things - accepting that critical stateful processes are indeed different from stateless processes and managing them accordingly - is still the best way. Getting away from this means getting away from MySQL and its ilk entirely. I hope we can get there one day; managing DB instances separately from your stateless instances is a pain in the ass.

MySQL only needs the my.cnf to start a server.

The permissions are stored along the database files (If i remember well), which is a separate problem. 1) It's only configured once on the master 2) Uber says that they don't reuse volumes [which implies many other consequences and special management around that].

By the way, Chef/Puppet/Ansible have build-in commands to manage that as well.

The unfortunate thing, in my opinion, is that so much of being a software/system administrator of any flavor is having been burned by that software. That experience makes you invaluable, but it also makes you expensive to hire. Expensive hires are really hard for startups to justify.

That said, if you reach the size where you're having to shard your DB to hit performance metrics, you're at the point where it will typically cost you less to absorb that expensive hire (or consultant) than to attempt to naively engineer around the problem.

https://www.percona.com/blog/

Or... startups are hard for experienced workers to justify.

Who would want to work for less money, shit equity, bad hours, on calls and surrounded by constant waves of variable level developers that you don't/can't get enough time to train.