News for hackers. Stuff that matters.

Ask HN: Transfer Large Amounts of Data

Hi HN,

My friend has folder on his computer with over 1.100 videos, in total it's about 300 GB.

He wants to hand me over the folder but I have no idea on how he should transfer everything over to my computer in the most efficient way possible.

My idea so far have been,

1. Just grab his harddrive and put it in my computer

2. Bring a portable hard drive over to his house

3. Rsync

4. Airtar / Wormhole

5. Torrent from his computer to mine

Do any of you have any better ideas?

( 12 comments | on HN )
Re:Ask HN: Transfer Large Amounts of Data by imagetic on Sunday December 11, 2022 @03:16PM (#33947002)
#1 https://massive.io while pricey per gb ($0.25/GB), you get the first 150GB free, so you'd be paying about $35 for the file transfer. Would be the least amount of effort of any of those scenarios, no real special tools required. Could even drag and drop the folder into the browser.

#2 would be portable hard drive. 300gb of video isn't enough to justify any syncing tools. Just drag and drop.

#3 rclone to Cloudflare R2 maybe, somewhere with no egress fees. Be about $5.

Re:Ask HN: Transfer Large Amounts of Data by imagetic on Sunday December 11, 2022 @03:18PM (#33947022)
And no wifi, ethernet for data on both sides. You want to saturate the pipe to get it quick. Don't let wifi signal strength be your speed factor.
Re:Ask HN: Transfer Large Amounts of Data by janalsncm on Tuesday December 06, 2022 @07:52PM (#33889311)
If you have access to a portable hard drive that’s pretty doable. Looking online [1] it seems that they can usually transfer at 30 mb/s which means it’ll only take about 3-4 hours.

[1] https://askubuntu.com/questions/41397/external-usb-hard-driv...

Re:Ask HN: Transfer Large Amounts of Data by Alifatisk on Tuesday December 06, 2022 @08:22PM (#33889571)
That sounds like the best option at the moment.
Re:Ask HN: Transfer Large Amounts of Data by anenefan on Tuesday December 06, 2022 @08:41PM (#33889724)
I found torrenting over even a local network was much slower than directly copying to a portable hard drive.

Most likely you want to settle on a process that verifies the copied files. If your friend uses M$ you could take a simple path and use a program like Terracopy [1] to be sure and just copy them all to a portable hard drive. (I've used it since working on a Win 10 system (whatever version it was) where quicker file copy was achieved by stepping back to iirc win 95 era and not verifying files.)

In linux there are many options - personally I tar folders up to 20 or 30 gigs using --verify (-W) ie tar -cvWf folder1 'folder1'/

... in your case you would need to direct the write of tar to the portable drive.

[1] https://codesector.com/teracopy

Re:Ask HN: Transfer Large Amounts of Data by Alifatisk on Tuesday December 06, 2022 @08:49PM (#33889776)
We're restricted to Windows sadly. But I'll check out Teracopy.
Re:Ask HN: Transfer Large Amounts of Data by version_five on Tuesday December 06, 2022 @08:10PM (#33889465)
What kind of constraints are you under, or what are you trying to optimize for? In many circumstances that wouldn't be considered a lot of data. Is low bandwidth the issue? If so, physical sharing the hard drive, or even mailing a usb hard drive makes sense. How valuable is the data and how much is your time worth?
Re:Ask HN: Transfer Large Amounts of Data by Alifatisk on Tuesday December 06, 2022 @08:22PM (#33889568)
I don't think none of us are in any constrains, we both have high-speed network (around 300 Mb/s down and 50 Mb/s up).

What I am trying to optimize for is the transfer rate, I would prefer this transfer takes less than a couple of hours.

The data is not that valuable, it would just be nice to have.

Re:Ask HN: Transfer Large Amounts of Data by version_five on Tuesday December 06, 2022 @08:38PM (#33889698)
Just try transfering it with sftp and see what kind of speed you get. Maybe use tar first to make a few big files, which will reduce overhead
Re:Ask HN: Transfer Large Amounts of Data by Alifatisk on Tuesday December 06, 2022 @08:48PM (#33889762)
Interesting, I'll add this to my list.
Re:Ask HN: Transfer Large Amounts of Data by sterlinm on Tuesday December 06, 2022 @08:57PM (#33889828)
Portable hard drive is probably the most practical. Another option not on this list would be rclone [1].

[1] https://rclone.org/docs/

Re:Ask HN: Transfer Large Amounts of Data by wazokazi on Tuesday December 06, 2022 @09:16PM (#33889935)
Take a gigabit network switch over to your friends house and connect both the machines to the same switch. Use windows to discover the shared folder on your friends computer and copy it to yours.