News for hackers. Stuff that matters.
Re:Rsync, GUIs, power, control, design, and decisions by klodolph on Tuesday July 09, 2019 @01:40PM (#20394501)
> Successful UNIX tools (or any tools, for the most part) are the ones with a simple concept manifested very thoroughly, and rsync is certainly that. [...] There are over 100 flags in rsync that you can select to do different things. So, this is what I mean by doing something simple, and doing it very well and very thoroughly.

Rsync is certainly not simple. It certainly doesn't do one thing. It's "extraordinarily versatile" to the point of madness. Here's what it does:

- Copies files from A to B

- Mirrors directories

- Computes diffs

- Applies patches

- Evaluates filters

- Runs as a daemon

- Runs over various remote shells

- Applies filters to filenames

And then it doesn't give you a good progress bar or let you resume interrupted operations, or gather errors at the end. I think this is a good case where the tool may have grown too large and unwieldy, and it may be better to replace it with an API or a collection of small, UNIX-like tools. For example, if I just want to copy files from point A to B, I can do this:

    tar c . | ssh host tar xv
If I want a progress bar, I can use pv,

    tar c . | pv | ssh host tar xv
Usually I find this easier to use than rsync.
Re:Rsync, GUIs, power, control, design, and decisions by sirn on Tuesday July 09, 2019 @01:49PM (#20394620)
I believe rsync can do two of your doesn't do list:

> And then it doesn't give you a good progress bar

    rsync --progress
> let you resume interrupted operations

    rsync --partial
...or enable both at the same time with rsync -P

(These features only available on samba.org's original rsync, and not OpenBSD openrsync[1], but I don't think openrsync has seen a widespread use yet, as it's still pretty new)

[1]: https://www.openrsync.org

Re:Rsync, GUIs, power, control, design, and decisions by klodolph on Tuesday July 09, 2019 @02:26PM (#20395060)
rsync --progress isn't very good, which is why I worded it that way.

If you use rsync --partial it has to rescan the source and destination directories again. All it does is let you resume transfers, and for some of us, transfers are not the slow part.

Re:Rsync, GUIs, power, control, design, and decisions by mturmon on Tuesday July 09, 2019 @02:13PM (#20394903)
Your "piped tar" doesn't cover the main use case of rsync, which is (unsurprisingly), remote file hierarchy sync'ing.

In particular, when there's a hierarchy on a remote machine that I want to sync to a partially-up-to-date hierarchy on a local machine, the required rsync command is a one-liner and it's very fast because it uses a diff.

Incidentally, the "tar" man page is not exactly a model of simplicity itself.

Re:Rsync, GUIs, power, control, design, and decisions by klodolph on Tuesday July 09, 2019 @02:32PM (#20395126)
> Your "piped tar" doesn't cover the main use case of rsync, which is (unsurprisingly), remote file hierarchy sync'ing.

That's exactly my point, isn't it? rsync does several different things, depending on how you invoke it, rather than being a UNIXy tool that does one thing well. Because the single tool does everything, you can't easily swap out different parts of it, and the single tool takes a hundred options.

Re:Rsync, GUIs, power, control, design, and decisions by aidenn0 on Tuesday July 09, 2019 @03:28PM (#20395657)
For copying files from A to B, I wrote a simple script[1]. rsync is not as universally available on hosts, the tar version is sometimes significantly faster than rsync when just copying, and rsync won't let me triangle-route copies.

https://gist.github.com/jasom/ecdec7767b5f774cd24aebba3c2ba3...

Re:Rsync, GUIs, power, control, design, and decisions by gchamonlive on Tuesday July 09, 2019 @01:51PM (#20394632)
What is simple in rsync is the idea of synchronising stuff from A to B. That is all it does in all the cases you mentioned.

It is hard to emulate what rsync does with archive mode enabled, which speeds up transfers, so tar isn't exactly a replacement for rsync.

There are better ways to show progress, for example using the flag `--info=progress2`. It doesn't behave well in automated scripts though

Re:Rsync, GUIs, power, control, design, and decisions by darkpuma on Tuesday July 09, 2019 @02:20PM (#20395000)
fyi, combining something like `--out-format="file: %f %l"` with `--progress` results in reasonably parse-able progress updates.

Keep in mind however that the format of --progress has changed in the past. Depending on your version of rsync it may or may not insert thousand seperating commas into the number of bytes written, e.g. 2,384,908 vs 2384908. The former can be supressed with --no-h, an option not available on at least some older versions of rsync that default to no commas.

Re:Rsync, GUIs, power, control, design, and decisions by klodolph on Tuesday July 09, 2019 @02:33PM (#20395143)
> It is hard to emulate what rsync does with archive mode enabled, which speeds up transfers, so tar isn't exactly a replacement for rsync.

Archive mode is what tar does by default. It is just an alias for -rlptgoD. How does this speed up transfers? Are you thinking of something else?

Re:Rsync, GUIs, power, control, design, and decisions by somacert on Tuesday July 09, 2019 @05:06PM (#20396591)
The "Does not resume interrupted operations" gripe confuses me because that is the main reason I reach for rsync, I have a large tree of files I need to be somewhere else and if the transfer fails I would like it to not transfer files already transferred.

So I use rsync, which offers, as a core feature, resuming interrupted operation.

Re:Rsync, GUIs, power, control, design, and decisions by Izkata on Wednesday July 10, 2019 @01:41PM (#20404349)
That one is badly phrased, it's about resuming a single interrupted large file. By default rsync will create a new temp file and start over, but there are options to make it resumable.
Re:Rsync, GUIs, power, control, design, and decisions by darkpuma on Tuesday July 09, 2019 @01:45PM (#20394558)
Adding pv to a tar|ssh pipeline is simpler than just passing -P to rsync? I must disagree.
Re:Rsync, GUIs, power, control, design, and decisions by klodolph on Tuesday July 09, 2019 @02:27PM (#20395072)
Maybe I wasn't clear about my complaint. My complaint is that rsync --progress sucks, I'm not complaining that it's hard to use.
Re:Rsync, GUIs, power, control, design, and decisions by darkpuma on Tuesday July 09, 2019 @02:32PM (#20395134)
I don't like the --no-h change they made to the --progress format (mentioned in another comment) because it broke my parsing once, but otherwise I don't really see a problem with it and I expect most people have never tried to parse it. What's so bad about --progress?

Edit: --progress also tells you how many files rsync has been told to sync, and of those, how many are remaining.

    2384908 100%   57.52MB/s    0:00:00 (xfr#1, to-chk=7/11)
e.g. transfer #1 out of 11 files or directories to check, with 7 remaining.
Re:Rsync, GUIs, power, control, design, and decisions by klodolph on Tuesday July 09, 2019 @02:37PM (#20395175)
--progress gives you only the progress of individual file transfers.
Re:Rsync, GUIs, power, control, design, and decisions by sirn on Tuesday July 09, 2019 @02:53PM (#20395334)
--info=progress2 actually gives you the progress of the whole transfer (but I don't think you can do both). You can use --info=name0,progress2 if you don't care about individual files scrolling the terminal.
Re:Rsync, GUIs, power, control, design, and decisions by klodolph on Tuesday July 09, 2019 @05:32PM (#20396858)
Seems like this user-hostile interface could have been averted if they gave rsync a "porcelain" mode, since --progress is frozen so it doesn't break scripts.
Re:Rsync, GUIs, power, control, design, and decisions by darkpuma on Tuesday July 09, 2019 @05:46PM (#20396996)
I think you're really exaggerating the severity of what is fundamentally just a minor matter of personal taste.

(I say this as somebody who's written a GUI wrapper around rsync to, among other things, implement my own progress bar. So I agree with the general principle of writing porcelain around rsync.) (Incidentally --progress has actually changed in a way that breaks (my) scripts in the past; the version that ships with MacOS and the latest version exhibit different behavior by default. It would be nice if there was a better way. If rsync followed the ffmpeg model of foremost being a library with a standard command line interface, I think that could be ideal.)