News for hackers. Stuff that matters.
Re:Jetpants: a MySQL toolkit for managing billions of rows a... by ralph on Friday June 08, 2012 @09:15AM (#4083986)
They're using pigz on [1] so they are compressing?

I thought it wasn't a great idea. OK for an on-the-fly solution to the problem but bittorrent or multicast would seem better; the serial route between machines isn't very fault tolerant requiring a start-from-scratch on failure.

socat > nc BTW, and does multicast.

As for ssh, it's a shame the "no encryption" option was removed.

Re:Jetpants: a MySQL toolkit for managing billions of rows a... by _b8r0 on Friday June 08, 2012 @09:18AM (#4083997)
Wow, I completely missed pigz in there for some reason and had to look it up.

I agree with you on all counts though!

Re:Jetpants: a MySQL toolkit for managing billions of rows a... by evanelias on Friday June 08, 2012 @09:40AM (#4084089)
re: fault tolerance, it's a fair point. Although in practice I've never had this fail part-way on me, and I've used it a couple hundred times with >600GB transfers.

We usually use this to copy to 2 or 3 machines at once; it's rare that we'd need to bring up 4+ slaves simultaneously, or split a shard into 4+ pieces. Most Linux distributions already have all the software needed except pigz, which is tiny and available in several packaging systems.

I'll definitely give socat a look though, thanks for the tip.

Re:Jetpants: a MySQL toolkit for managing billions of rows a... by ralph on Friday June 08, 2012 @07:04PM (#4086545)
Oh, OK, I agree, for 2-3 machines reliability isn't an issue, I was thinking more dozens.