News for hackers. Stuff that matters.
Re:Ask HN: I need to transfer 10TB of Data across Pacific by cat9 on Sunday June 28, 2015 @10:31PM (#9796119)
The best solution depends entirely on your optimization conditions.

If transaction time is important, hard drives are high bandwidth. Ten 1TB hard drives are $500. Two 6TB hard drives are also $500, but you have fewer drives to worry about shipping. Five 2TB hard drives are about $370, if drive price is more important. Using ten 1TB drives is only best if we're optimizing for load / unload time and have that degree of parallelism in the pipeline. Otherwise, one of the other two will be better, depending on price sensitivity vs. other factors.

If it's not as time-sensitive, something like BitTorrent Sync will get the job done. This doesn't require shipments, additional hardware, or mucking about with hardware at either end, so it's very nice if "get it there ASAP" isn't a constraint.

Regardless, don't ship the only copy.

Re:Ask HN: I need to transfer 10TB of Data across Pacific by stkim1 on Sunday June 28, 2015 @11:17PM (#9796235)
Thanks for the concise summary.
Re:Ask HN: I need to transfer 10TB of Data across Pacific by bogrollben on Sunday June 28, 2015 @10:04PM (#9796064)
Mail it to me at the hacker house in Kansas City. We do uploads for people all the time using Google Fiber. Probably take a few hours (YMMV). Google Homes for Hackers. My cell is right there on the site.
Re:Ask HN: I need to transfer 10TB of Data across Pacific by namecast on Sunday June 28, 2015 @10:35PM (#9796129)
+1 for being awesome - but the OP mentioned 'across the Pacific' - if he's got to transfer 10TB to, say, Sydney or Beijing, I'd bet your upstream won't be the limiting factor, it'll be the congested peering ports and high latency links between the States and APAC providers.

That said, nothing ventured, nothing gained. Maybe try a speed test between the hacker house and the intended AS before anyone ships a drive anywhere, and see if the other end can keep up with you?

Re:Ask HN: I need to transfer 10TB of Data across Pacific by stkim1 on Monday June 29, 2015 @09:50AM (#9798017)
I really appreciate your warm gesture. I wish my neighbors are just as kind as you are, sir.
Re:Ask HN: I need to transfer 10TB of Data across Pacific by tantalor on Sunday June 28, 2015 @10:37PM (#9796133)
Re:Ask HN: I need to transfer 10TB of Data across Pacific by stkim1 on Sunday June 28, 2015 @11:22PM (#9796257)
See? this is funny. Some says this is the age of data. 10T is just a tiny amount of bits for some entities, and I cannot reliably transmit that much across pacific other than shipping actual HDD. :(

Thank you for the comic.

Re:Ask HN: I need to transfer 10TB of Data across Pacific by toast0 on Sunday June 28, 2015 @10:39PM (#9796136)
Two 6 TB or three 4 TB is probably a better choice than ten 1 TB. Although you may want to have an extra disk so one could totally fail in transit; so maybe six 2 TB. Build a big parity archive (par2 or similar) first. If you have 1gbps end to end, it would take most of a day (make sure you have big TCP windows)
Re:Ask HN: I need to transfer 10TB of Data across Pacific by laurencerowe on Sunday June 28, 2015 @10:49PM (#9796166)
With long ping times you want to split your upload into many parallel parts so the odd dropped packet doesn't bring things to a halt. Then it just depends on your bandwidth.

We were able to saturate the gigabit nic on an ec2 instance in Ireland uploading to S3 in Oregon. At that rate a 10TB will take a little over a day.

Re:Ask HN: I need to transfer 10TB of Data across Pacific by georgerobinson on Monday June 29, 2015 @06:08PM (#9801491)
I'm puzzled. Can you explain what you mean by bringing to a halt?

Do you mean a dropped packet might terminate the TCP connection; or at worst result in the timeout of a segment and a return to slow start? Nevertheless, I would have thought a timeout is a rare event at best, even on high latency links, due to the post-Jacobsen RTT estimator and fast retransmit kicking in at 3 duplicate ACKs?

Re:Ask HN: I need to transfer 10TB of Data across Pacific by gesman on Sunday June 28, 2015 @09:43PM (#9796011)
5 x 2TB HD's bought at Costco + shipping charges.

Total cost: <$1k

Rather small package. "Upload" time <10 days.

Re:Ask HN: I need to transfer 10TB of Data across Pacific by codeonfire on Monday June 29, 2015 @03:54AM (#9796880)
Whether you can use distributed web services like s3, elastic mapreduce, or google storage to copy it largely depends on what kind of ISP connection you have on both ends. With gigabit and multipart transfers it seems feasible within a week.
Re:Ask HN: I need to transfer 10TB of Data across Pacific by drallison on Monday June 29, 2015 @09:38AM (#9797951)
Sneakernet (https://en.wikipedia.org/wiki/Sneakernet) is likely the best choice.
Re:Ask HN: I need to transfer 10TB of Data across Pacific by dgomesbr on Sunday June 28, 2015 @09:23PM (#9795946)
Bit torrent it to the other side of the pacific. With a 35mbps connection will take like a month.
Re:Ask HN: I need to transfer 10TB of Data across Pacific by meat_fist on Sunday June 28, 2015 @09:32PM (#9795978)
Mail a HD.
Re:Ask HN: I need to transfer 10TB of Data across Pacific by 001sky on Sunday June 28, 2015 @09:56PM (#9796049)
FedEx or similar
Re:Ask HN: I need to transfer 10TB of Data across Pacific by deskinner on Sunday June 28, 2015 @09:51PM (#9796036)
Globus.