News for hackers. Stuff that matters.
Re:Gcloud storage: up to 94% faster data transfers for Cloud... by rsync on Friday October 07, 2022 @03:30PM (#33125854)
From the article:

"The new gcloud storage CLI offers significant performance improvements over the existing gsutil ..."

It makes sense to see this comparison because 'gsutil' was relatively poor - we[1] made a decision to deprecate the 'gsutil' binary[2] in our environment in favor of rclone[3] because rclone is better in every way.

We made the same decision for 's3cmd'.

All of that to say:

The biggest question to ask about this is "how does it compare to rclone". I suspect the answer is that authentication and token handling (and things like that) are much better as rclone has a pretty clunky auth workflow for google cloud resources. That is not the case for Amazon resources which behave just as you'd expect them to, with regard to API tokens, and make it hard to justify using any tool other than rclone.

[1] rsync.net

[2] Yes, a binary - we "freeze" python into binary exe because we don't allow interpreters in our environment.

[3] https://rclone.org/

Re:Gcloud storage: up to 94% faster data transfers for Cloud... by xen2xen1 on Friday October 07, 2022 @03:55PM (#33126154)
Rclone is clunky because it works with everything, even things like OneDrive on Linux, which is not supported officially. Love rclone. Also just a great copy / move utility.
Re:Gcloud storage: up to 94% faster data transfers for Cloud... by kldx on Friday October 07, 2022 @03:58PM (#33126197)
How do you freeze python tools into binaries?
Re:Gcloud storage: up to 94% faster data transfers for Cloud... by kleton on Friday October 07, 2022 @04:05PM (#33126270)
par- I assume this is the open sourced version https://github.com/google/subpar
Re:Gcloud storage: up to 94% faster data transfers for Cloud... by rsync on Friday October 07, 2022 @05:34PM (#33127138)
"How do you freeze python tools into binaries?"

I think there are several workflows for doing this - I know in the past we used something called 'cx_Freeze' but I think the person in charge of this is now using 'py2exe' ?

I think the 'borg' project packages up their single file .exe distribution with 'pyinstaller' but I may not fully understand that part of their process...

Re:Gcloud storage: up to 94% faster data transfers for Cloud... by icelancer on Saturday October 08, 2022 @02:54AM (#33130373)
I use py2exe with pretty good success in deployments at my company. It's bloated and such but sometimes it's just the easiest path forward.
Re:Gcloud storage: up to 94% faster data transfers for Cloud... by judge2020 on Friday October 07, 2022 @08:18PM (#33128572)
Pyinstaller is the most actively maintained and in-use, but nuitka is on the rise as well. There are less-active but still serviceable ones like py2exe as well

https://pyinstaller.org/en/stable/usage.html

https://www.nuitka.net/

Re:Gcloud storage: up to 94% faster data transfers for Cloud... by faizshah on Friday October 07, 2022 @05:20PM (#33126997)
I use s5cmd, it’s a fast parallel s3 cli client: https://github.com/peak/s5cmd

Haven’t heard of rclone before so can’t compare yet.

Re:Gcloud storage: up to 94% faster data transfers for Cloud... by neuronexmachina on Friday October 07, 2022 @08:33PM (#33128682)
I'd also be curious about how rclone's parallelization compares to gcloud storage's in practice:

> Second, it utilizes a new parallelization strategy that treats task management as a graph problem, which allows more work to be done in parallel with far less overhead.

> ... In gcloud storage, all operations happen in parallel. As an example, parallel composite uploads are enabled automatically based on bucket configuration. This is a vast improvement compared to gsutil, which requires the -m (parallel operations) flag to improve the performance for uploads and downloads.

Re:Gcloud storage: up to 94% faster data transfers for Cloud... by xani_ on Friday October 07, 2022 @09:50PM (#33129097)
how faster is rclone compared to s3cmd ? We're using s3cmd to sync some stuff for backup purpose (many many small-ish) files and performance has been generally mediocre
Re:Gcloud storage: up to 94% faster data transfers for Cloud... by tebbers on Saturday October 08, 2022 @01:48AM (#33130099)
We often have to sync hundreds of thousands of files to S3 and we use this which often hits 300 files a second on a 1Gbps link:

https://github.com/mishudark/s3-parallel-put

Re:Gcloud storage: up to 94% faster data transfers for Cloud... by TheReveller on Saturday October 08, 2022 @12:02AM (#33129705)
How are gcp and aws different in terms of auth required by rclone?
Re:Gcloud storage: up to 94% faster data transfers for Cloud... by flanked-evergl on Saturday October 08, 2022 @01:50AM (#33130116)
I have also had persistent issues with gsutil and have used rclone instead, rclone is clunky but it has been an ever increasing part of our toolkit and I don't see that changing soon.