Dockerizing MySQL at Uber Engineering

Re:Dockerizing MySQL at Uber Engineering by johnkoepi on Monday November 28, 2016 @11:54AM (#13056641)

> 3) Transferring data from one host to another when you need to bring up a stateful process does not scale. Above a few gigs, the transfer process creates significant load on both the source and destination host, and will easily saturate the link between the two...

It depends. @falcolas, could you point out a few other solutions then that? (for resharding live, other then replication / copy / rsync / ...)

[ Parent | on HN ]

Re:Dockerizing MySQL at Uber Engineering by falcolas on Monday November 28, 2016 @12:06PM (#13056728)

Copying data is always going to be expensive, but it can't be avoided.

The lightest weight solution I've seen is restoring from a daily backup in something like S3, then setting up as a slave from a live master to catch up on the day's binlogs. Still a lot of data to move and load, but at least it's not the entire contents of the DB.

The best you can do is be in control of when data transfers happens so you're doing it when it makes sense and not in the middle of your highest traffic period (which is what frequently happens when attempting to automatically scale DBs in response to load).

[ Parent | on HN ]

Re:Dockerizing MySQL at Uber Engineering by seschwar on Tuesday November 29, 2016 @04:15AM (#13061847)

That sounds a bit like what Joyent is doing in their Autopilot Pattern implementation for MySQL: https://www.joyent.com/blog/dbaas-simplicity-no-lock-in

[ Parent | on HN ]

Re:Dockerizing MySQL at Uber Engineering by user5994461 on Monday November 28, 2016 @12:30PM (#13056916)

I'd add that one can use throttling to bring new databases in live, while sustaining peak traffic.

Some databases have a configurable limit in MB/s for replication, or it's possible to assign disk/CPU quota on the slave to slow it down.

Combine that with good planning and monitoring, you'll be fine =)

[ Parent | on HN ]