News for hackers. Stuff that matters.
Re:Jetpants: a MySQL toolkit for managing billions of rows a... by _b8r0 on Friday June 08, 2012 @04:42AM (#4083332)
It looks interesting but one small thing. I looked into the transferring large files quickly link[1] and saw that they were using netcat and tar to transfer files. This is not necessarily optimal[2], and applying some compression can go a long way, although this will be dependent on the use case. Compression also has the added bonus of transferring less data across the network and (if you don't uncompress) less space at the other end.

SSH is also an option (a slow option, but an option) that provides certain things (encryption, authentication) that make it ideally suited for transfers across network boundaries.

[1] - http://engineering.tumblr.com/post/7658008285/efficiently-co...

[2] - http://www.ndchost.com/wiki/server-administration/netcat-ove...

Re:Jetpants: a MySQL toolkit for managing billions of rows a... by ralph on Friday June 08, 2012 @09:15AM (#4083986)
They're using pigz on [1] so they are compressing?

I thought it wasn't a great idea. OK for an on-the-fly solution to the problem but bittorrent or multicast would seem better; the serial route between machines isn't very fault tolerant requiring a start-from-scratch on failure.

socat > nc BTW, and does multicast.

As for ssh, it's a shame the "no encryption" option was removed.

Re:Jetpants: a MySQL toolkit for managing billions of rows a... by _b8r0 on Friday June 08, 2012 @09:18AM (#4083997)
Wow, I completely missed pigz in there for some reason and had to look it up.

I agree with you on all counts though!

Re:Jetpants: a MySQL toolkit for managing billions of rows a... by evanelias on Friday June 08, 2012 @09:40AM (#4084089)
re: fault tolerance, it's a fair point. Although in practice I've never had this fail part-way on me, and I've used it a couple hundred times with >600GB transfers.

We usually use this to copy to 2 or 3 machines at once; it's rare that we'd need to bring up 4+ slaves simultaneously, or split a shard into 4+ pieces. Most Linux distributions already have all the software needed except pigz, which is tiny and available in several packaging systems.

I'll definitely give socat a look though, thanks for the tip.

Re:Jetpants: a MySQL toolkit for managing billions of rows a... by ralph on Friday June 08, 2012 @07:04PM (#4086545)
Oh, OK, I agree, for 2-3 machines reliability isn't an issue, I was thinking more dozens.