Network file transfer with on-the-fly compression

Posted by codemechanic on Wednesday May 06, 2009 @01:26PM
from the codelathe.com dept.

http://www.codelathe.com/blog/index.php/2009/04/09/network-file-transfer-with-on-the-fly-compression/

( 6 comments | 9 points | on HN )

Re:Network file transfer with on-the-fly compression by wmf on Wednesday May 06, 2009 @01:59PM (#596322)

I'd like to see a comparison between this and scp with compression.

[ Parent | on HN ]

Re:Network file transfer with on-the-fly compression by KirinDave on Wednesday May 06, 2009 @02:24PM (#596407)

That's the first thing I thought of too. The -C flag is there specifically for this very scenario.

Also, would "j" have worked better than "z" in the tar line? bzip2 is much slower than gzip, but does a better job.

[ Parent | on HN ]

Re:Network file transfer with on-the-fly compression by wmf on Wednesday May 06, 2009 @04:05PM (#596612)

Really we should have adaptive compression so that the CPU and network usage is balanced. Too bad it may require "rampant layering violation" to implement.

[ Parent | on HN ]

Re:Network file transfer with on-the-fly compression by doosra on Wednesday May 06, 2009 @02:38PM (#596433)

scp -C does the same thing. Both methods use gzip for compression.

[ Parent | on HN ]

Re:Network file transfer with on-the-fly compression by rarrrrrr on Wednesday May 06, 2009 @03:13PM (#596509)

But the tar method will generally be 5-15x faster for large collections of small files.

Handing it with tar makes operations on both ends asynchronous. scp (or rsync, etc.) of multiple files does file opens on each end in lockstep, resulting in a lot of idle wait time.

[ Parent | on HN ]

Re:Network file transfer with on-the-fly compression by mwcremer on Wednesday May 06, 2009 @06:17PM (#596856)

Note that this is not likely to help with media files (mp3, mpeg, jpeg).

[ Parent | on HN ]