News for hackers. Stuff that matters.
Re:The bandwidth of a Boeing 747 and its impact on web browsing by slyall on Thursday June 21, 2012 @07:28AM (#4141423)
There are actually special file transfer protocols for high-bandwidth high-latency situations where you want to transfer gigabytes/terabytes across the world.

The main example I've heard is special effects companies like Weta Digital in New Zealand wanting to deliver video to the US across a 200ms latency link.

There are a few open protocols in various stages of development or you can pay for a product like FileCatalyst http://www.filecatalyst.com/ ( some interesting case studies on their website ).

Re:The bandwidth of a Boeing 747 and its impact on web browsing by mevodig on Thursday June 21, 2012 @09:20AM (#4141824)
Yes, you're likely thinking about the Aspera range of file transfer software which is quite popular within the film industry.

It's actually quite elegant; a control connection over SSH plus a custom UDP protocol (FASP) for data. It does work as advertised and has some nice features that are handy in production, but licensing is very expensive ($20K for point-to-point at 1gb/s).

Re:The bandwidth of a Boeing 747 and its impact on web browsing by efuquen on Thursday June 21, 2012 @10:21AM (#4142088)
"There are a few open protocols in various stages of development"

Can you, or somebody else please name some? It always irks me a little when someone just mentions that "these projects" exist, and then fail to mention any of them.

Re:The bandwidth of a Boeing 747 and its impact on web browsing by EvanAnderson on Thursday June 21, 2012 @10:58AM (#4142294)
We had a discussion about this over on Server Fault (http://serverfault.com/q/111813/7200) and UFTP (http://www.tcnj.edu/~bush/uftp.html), UDT (http://udt.sourceforge.net/), and Tsunami-UDP (http://tsunami-udp.sourceforge.net/) all came up as viable open source offerings (along with FileCatalyst and Aspera in the commercial space).
Re:The bandwidth of a Boeing 747 and its impact on web browsing by Terretta on Thursday June 21, 2012 @05:21PM (#4144254)
No need to over engineer...

We offer a custom protocol file transfer as a service (uses UDP instead of TCP), but you can do a home grown solution yourself very easily.

If you have a big file of compressed content (video), do the following. If you have a lot of small files, tar and gzip them, then do the following.

    md5 your file
    split your file into chunks (eg parts _a - _z, or _aa - _zz)
    use ncftp or similar capable of parallel transfers
    optimize between 8 and 30 parallel depending on latency to saturate link
    concat the file at the other end
    check the md5
You can handily saturate a link from EU to Australia or India to US this way.

That said, for Aussie customers sending us 1000s of movies, we FedEx or DHL pelican cases full of drives.

Re:The bandwidth of a Boeing 747 and its impact on web browsing by Karunamon on Thursday June 21, 2012 @09:18AM (#4141814)
Aspera (http://asperasoft.com) is another one (but my god are they expensive.. the most basic setup possible will run you about $25K). I'm definitely interested if there are open alternatives.
Re:The bandwidth of a Boeing 747 and its impact on web browsing by smithian on Thursday June 21, 2012 @08:22AM (#4141602)
We use Filecatalyst to transfer large files internally and to/from customers. +1 works as advertised. What open protocols were you thinking of?