gperciva
Tarsnap's deduplication code is not available under an open-source license.
To print progress with tarsnap 1.0.39, send it a SIGUSR1 or SIGINFO. On FreeBSD, you can do this by pressing ctrl-t. On Linux, you have to use the unfortunately-named `kill` or `killall` command, such as
killall -SIGUSR1 tarsnap
https://www.tarsnap.com/tips.html#check-current
(Note that Tarsnap is not responsible for naming the unix `kill` or `killall` commands.)
In the unreleased git version of tarsnap, there's a `--progress-bytes SIZE` command, which prints a progress message after every SIZE bytes are processed.
As a general note: the tarsnap-users mailing list is a great place to ask for tips. As you mentioned in your lessons learned, some of the options could have helped a lot (such as `--recover`) https://www.tarsnap.com/lists.html
(Disclaimer: I'm employed by Tarsnap Backup Inc.)
The command-line binary generally takes "max time" and "max ram" (as percent and/or raw value) and estimates appropriate cost values. As of version 1.3.1, you can manually specify cost values for the binary if you want.
Main page, including the signed tarball: http://www.tarsnap.com/scrypt.html