Re:Jetpants: a MySQL toolkit for managing billions of rows a...
by ralph on Friday June 08, 2012 @09:15AM
(#4083986)
They're using pigz on [1] so they are compressing?
I thought it wasn't a great idea. OK for an on-the-fly solution to the problem but bittorrent or multicast would seem better; the serial route between machines isn't very fault tolerant requiring a start-from-scratch on failure.
socat > nc BTW, and does multicast.
As for ssh, it's a shame the "no encryption" option was removed.
Re:Jetpants: a MySQL toolkit for managing billions of rows a...
by _b8r0 on Friday June 08, 2012 @09:18AM
(#4083997)
Wow, I completely missed pigz in there for some reason and had to look it up.
I agree with you on all counts though!
Re:Jetpants: a MySQL toolkit for managing billions of rows a...
by evanelias on Friday June 08, 2012 @09:40AM
(#4084089)
re: fault tolerance, it's a fair point. Although in practice I've never had this fail part-way on me, and I've used it a couple hundred times with >600GB transfers.
We usually use this to copy to 2 or 3 machines at once; it's rare that we'd need to bring up 4+ slaves simultaneously, or split a shard into 4+ pieces. Most Linux distributions already have all the software needed except pigz, which is tiny and available in several packaging systems.
I'll definitely give socat a look though, thanks for the tip.
SSH is also an option (a slow option, but an option) that provides certain things (encryption, authentication) that make it ideally suited for transfers across network boundaries.
[1] - http://engineering.tumblr.com/post/7658008285/efficiently-co...
[2] - http://www.ndchost.com/wiki/server-administration/netcat-ove...