Re:Jetpants: a MySQL toolkit for managing billions of rows a...
by _b8r0 on Friday June 08, 2012 @09:18AM
(#4083997)
Wow, I completely missed pigz in there for some reason and had to look it up.
I agree with you on all counts though!
Re:Jetpants: a MySQL toolkit for managing billions of rows a...
by evanelias on Friday June 08, 2012 @09:40AM
(#4084089)
re: fault tolerance, it's a fair point. Although in practice I've never had this fail part-way on me, and I've used it a couple hundred times with >600GB transfers.
We usually use this to copy to 2 or 3 machines at once; it's rare that we'd need to bring up 4+ slaves simultaneously, or split a shard into 4+ pieces. Most Linux distributions already have all the software needed except pigz, which is tiny and available in several packaging systems.
I'll definitely give socat a look though, thanks for the tip.
I thought it wasn't a great idea. OK for an on-the-fly solution to the problem but bittorrent or multicast would seem better; the serial route between machines isn't very fault tolerant requiring a start-from-scratch on failure.
socat > nc BTW, and does multicast.
As for ssh, it's a shame the "no encryption" option was removed.