Re:Moving 6 Billion Messages Without Being Noticed
by banksyw on Monday February 27, 2012 @05:56AM
(#3638533)
Disclaimer: I wrote that article.
Percona toolkit has some useful stuff. Thanks for the link.
In this case our solution ended up being pretty much the same as pt-table-sync except that it made use of our distributed task queuing system so was MUCH faster than if we had used a single threaded perl script. (Up to a few hundred chunks of rows being checksummed/synced in parallel.) I didn't talk about this aspect directly in the article to keep it focused.
Not sure if it would be possible to use pt-table-sync in that way, certainly would have taken a lot more work to get a third-party perl script to distribute the work through our job queue.
Percona toolkit has some useful stuff. Thanks for the link.
In this case our solution ended up being pretty much the same as pt-table-sync except that it made use of our distributed task queuing system so was MUCH faster than if we had used a single threaded perl script. (Up to a few hundred chunks of rows being checksummed/synced in parallel.) I didn't talk about this aspect directly in the article to keep it focused.
Not sure if it would be possible to use pt-table-sync in that way, certainly would have taken a lot more work to get a third-party perl script to distribute the work through our job queue.