News for hackers. Stuff that matters.

A comment on Rclone – Sync files and directories to many cloud storage providers.

Re:Rclone – Sync files and directories to many cloud storage... by gary-kim on Monday April 06, 2020 @04:21AM (#22791801)
I'm an occasional contributor for Rclone. Rclone holds a special place for me as the first open source project that I got seriously involved in and made me far more comfortable with working in open source.

Rclone is now very prevalent in my infrastructure. Almost all my websites are updated by a CI job that builds the website from a repo and pushes it up to the hosting server. There's an encrypted Rclone config in the repo and the password for it is a really long randomly generated string that gets saved in the CI as a secret. Rclone with Restic is how most of my servers get backed up, Rclone is how I access my Nextcloud and Google Drive, I have a containerized s3 compatible storage system that actually stores it's data on a Rclone remote (I hope `serve s3` gets implemented soon this setup can be simpler [1]), and much more.

I'm using it so much that I'm running up against the Google Drive API limit even though I'm using my own key.

* [1] https://github.com/rclone/rclone/issues/3382

[ Parent | on HN ]
Re:Rclone – Sync files and directories to many cloud storage... by aspenmayer on Monday April 06, 2020 @12:19PM (#22794981)
You may benefit from rotating service accounts to sidestep the API limits. I haven't implemented this on my own setup yet, but I know that it is scriptable. Implementation differences prevent one overarching tutorial, but I left some breadcrumbs for you below.

https://github.com/l3uddz/cloudplow

https://github.com/Rhilip/AutoRclone

https://hub.docker.com/r/hotio/rclone

[ Parent | on HN ]
Re:Rclone – Sync files and directories to many cloud storage... by DelightOne on Monday April 06, 2020 @11:33AM (#22794605)
Is it better to integrate with the go-api or to write files to a local folder and run rclone on that to upload files?

I'm planning to upload files from my app and let an rclone cronjob occasionally sync between my backends.

[ Parent | on HN ]
Re:Rclone – Sync files and directories to many cloud storage... by leotaku on Monday April 06, 2020 @08:53AM (#22793262)
I did not know that Restic was built on top of Rclone, but of course that makes a lot of sense. Thanks for contributing to a project that has made my life much easier!
[ Parent | on HN ]
Re:Rclone – Sync files and directories to many cloud storage... by tankenmate on Monday April 06, 2020 @09:51AM (#22793668)
rclone is just one of the backends you can use for restic. personally i find it far more performant to use a local disk for your restic repo (and keep the desired number of snapshots, etc there) and then use rclone to clone that repo to a cloud provider. personally i used backblaze's b2 api as it is generally cheaper.

restic is somewhat sensitive to the latency of the backend it uses.

[ Parent | on HN ]
Re:Rclone – Sync files and directories to many cloud storage... by anderspitman on Monday April 06, 2020 @11:03AM (#22794347)
I do the exact same thing.
[ Parent | on HN ]