News for hackers. Stuff that matters.
Re:Moving 25TB data from one S3 bucket to another took 7 eng... by rsync on Monday September 14, 2020 @01:18AM (#24467017)
Are there any cloud storage providers that have 'rclone' built into their environment ?

So I could, for instance:

  ssh [email protected] rclone s3:/some/bucket gdrive:/blah
or:

  ssh [email protected] rclone s3:/source/files /my/rsync.net/account
Well I'll be damned! It appears that there is such a provider :

https://www.rsync.net/products/rclone.html

https://www.rsync.net/products/universal.html

Re:Moving 25TB data from one S3 bucket to another took 7 eng... by brylie on Monday September 14, 2020 @01:40AM (#24467137)
> NO Charges for ingress/egress

Is this too good to be true?!

Thank you for sharing! Yet another gem found in HN comments.

Re:Moving 25TB data from one S3 bucket to another took 7 eng... by rsync on Monday September 14, 2020 @02:00AM (#24467244)
Don't thank us, thank Hurricane Electric (he.net).

20 years ago, I was on irc.lightning.net and thought that was the best run efnet server. The MOTD advertised their IP transit services. They became he.net and that is why rsync.net does (most) of their IP transit with them.

Re:Moving 25TB data from one S3 bucket to another took 7 eng... by throwaway9d0291 on Monday September 14, 2020 @04:41AM (#24468090)
No, the tradeoff is that even with volume, rsync.net storage costs are $15/TBmonth [0], which is ~3x managed alternatives [1,2] and ~10x DIY alternatives [3]. To be fair, ZFS snapshots are "free" but whether that's worth 3x cost will depend on your data.

The lack of bandwidth charges sounds really nice but [1,3] have that too and unlike [1,3], Rsync.net doesn't support protocols that are suited for bandwidth-intensive applications (like serving large files to end-users). It only supports SSH-based transfers [4].

[0]: https://www.rsync.net/pricing.html

[1]: https://wasabi.com/cloud-storage-pricing/

[2]: https://www.backblaze.com/b2/cloud-storage-pricing.html

[3]: https://www.hetzner.de/dedicated-rootserver/matrix-sx?countr...

[4]: https://www.rsync.net/products/platform.html

Re:Moving 25TB data from one S3 bucket to another took 7 eng... by rsync on Monday September 14, 2020 @12:47PM (#24472259)
It is true that our service is more expensive.

If you look at the pricing on that rclone page[1] you'll see that at quantity it is as low as 0.5 Cents Per GB / Month.

As for bandwidth intensive protocols, we have HPN-SSH[2] patches built into our environment which allow for high bandwidth transfers, over SSH, over long WAN links.

[1] https://www.rsync.net/products/rclone.html

[2] https://www.psc.edu/hpn-ssh

Re:Moving 25TB data from one S3 bucket to another took 7 eng... by throwaway9d0291 on Tuesday September 15, 2020 @03:03AM (#24478590)
> If you look at the pricing on that rclone page[1] you'll see that at quantity it is as low as 0.5 Cents Per GB / Month.

That's good but at the volume needed to reach it, I imagine that you're in the realm of "contact us and get a special deal" at other providers (e.g. AWS, GCP, Azure) too.

Plus, we haven't even got to the "you have to pay for capacity, not usage" aspect of rsync.net pricing.

Pricing is the big thing I wish you'd fix. Your service has a bunch of cool features but they're just not worth that price, not when I can get most of the same features from Hetzner for a fraction of the cost.

> As for bandwidth intensive protocols, we have HPN-SSH[2] patches built into our environment which allow for high bandwidth transfers, over SSH, over long WAN links.

That's cool (honestly, not being condescending) but my point was that for most of the reasons you'd be excited about free bandwidth, rsync.net isn't a viable solution. For example you can't host images, videos, software updates, machine learning datasets or other large binaries and serve them over HTTP on the public internet.

Re:Moving 25TB data from one S3 bucket to another took 7 eng... by rsync on Thursday September 17, 2020 @12:09AM (#24501103)
"rsync.net isn't a viable solution. For example you can't host images, videos, software updates, machine learning datasets or other large binaries and serve them over HTTP on the public internet."

Correct. We don't do these things and we never will.

When you 'nmap' an rsync.net storage array, you get:

22 TCP

... and that's it. There are no other services running, or offered. There are no interpreters in the environment. The filesystems are mounted noexec,nosuid.

We do one very simple thing and that's it.

Re:Moving 25TB data from one S3 bucket to another took 7 eng... by throwaway9d0291 on Thursday September 17, 2020 @05:11AM (#24502565)
And were you to do that simple thing at a competitive price, it'd be great.

But you don't.

To be competitive you either need to drop prices or do more stuff to be worth the money.

Re:Moving 25TB data from one S3 bucket to another took 7 eng... by mekster on Sunday September 20, 2020 @09:06PM (#24538813)
To be fair, storage cost is their only pricing, meaning lowering it will have a great impact on their income when others charge for bandwidth and extra support.

But having no other cost than the storage itself is relieving and having the service running for nearly 20 years gives you a peace of mind on their reliability. I also hear their support is good

I would want a bit more modern web interface though. BorgBase does look interesting to me, except only 2 years of operation can't tell about its reliability and longevity.

Re:Moving 25TB data from one S3 bucket to another took 7 eng... by chromedev on Monday September 14, 2020 @02:40AM (#24467452)
Google Cloud doesn't charge for egress to Google Drive as well. If you use GCP to transfer files to Google Drive via rclone, then you can download them elsewhere without any charges.