News for hackers. Stuff that matters.
Re:dd built-in progress introduced in coreutils 8.24 by brynet on Sunday January 10, 2016 @09:41PM (#10878446)
On BSD, unlike Linux, many utilities including dd(1) will dump a progress report when receiving SIGINFO. This was traditionally bound to ^T in shells, either by default or using stty(1).
Re:dd built-in progress introduced in coreutils 8.24 by justin_vanw on Sunday January 10, 2016 @10:11PM (#10878544)
It's SIGUSR1 on Linux, however there is no signal bound to ^T in Linux, nor is it apparently possible to bind new ones.

`killall -USR1 dd` works like a charm, or for the impatient: watch killall -USR1 dd

Re:dd built-in progress introduced in coreutils 8.24 by oxplot on Monday January 11, 2016 @02:20AM (#10879338)
This example highlights a very useful alternative function of `watch`: to simply run a command on regular intervals without caring about the output.
Re:dd built-in progress introduced in coreutils 8.24 by agnivade on Tuesday January 12, 2016 @10:49PM (#10892210)
Yea, but its a pain to open another tab and run another command and switch back to this tab to see the progress.

Why they didn't give a progress bar in the first place beats me.

Re:dd built-in progress introduced in coreutils 8.24 by echochar on Sunday January 10, 2016 @10:51PM (#10878674)
progress(1) is a separate utility on BSD; one of my favorites. (Although it is also built in to BSD ftp.)

I regularly use progress with dd, tar and other utilities.

At least one BSD dd now has a "msgfmt" option that outputs parseable, printf-style format strings.

Truthfully, I prefer it when these basic utilities are static except for fixing bugs. Using the new features in scripts means the scripts might lose some portability.

Re:dd built-in progress introduced in coreutils 8.24 by crishoj on Monday January 11, 2016 @12:16AM (#10878964)
There's also the pv(1) utility on Linux and OS X Homebrew:

  DESCRIPTION
  
       pv  shows the progress of data through a pipeline by giving infor-
       mation such as time elapsed, percentage completed  (with  progress
       bar), current throughput rate, total data transferred, and ETA.

       To use it, insert it in a pipeline between two processes, with the
       appropriate options.  Its standard input will be passed through to
       its  standard output and progress will be shown on standard error.
(edit: formatting)
Re:dd built-in progress introduced in coreutils 8.24 by whalesalad on Sunday January 10, 2016 @11:01PM (#10878707)
I use this often on OS X :)
Re:dd built-in progress introduced in coreutils 8.24 by cnvogel on Monday January 11, 2016 @02:36AM (#10879396)
This is how it looks...

    [sc-lx-phys-05 /home/cvogel]
    $ dd if=/dev/zero of=/dev/null status=progress
    11086075904 bytes (11 GB) copied, 7.000001 s, 1.6 GB/s
Re:dd built-in progress introduced in coreutils 8.24 by Luyt on Sunday January 10, 2016 @09:29PM (#10878408)
I find this style of formatting specifiers an eyesore:

    fprintf (stderr,
            _("%"PRIuMAX"+%"PRIuMAX" records in\n"
              "%"PRIuMAX"+%"PRIuMAX" records out\n"),
            r_full, r_partial, w_full, w_partial);
Re:dd built-in progress introduced in coreutils 8.24 by LukeShu on Sunday January 10, 2016 @09:57PM (#10878496)
I assume you mean the `%"PRIuMAX"` bits. It is an eyesore, but it is how to portably write the specifiers for types whose size my vary across architectures. That is, blame POSIX, not GNU coding style.
Re:dd built-in progress introduced in coreutils 8.24 by _kst_ on Sunday January 10, 2016 @10:16PM (#10878563)
The `PRIuMAX` et al macros are specified by ISO C (starting with the 1999 standard), not (just) by POSIX.
Re:dd built-in progress introduced in coreutils 8.24 by LukeShu on Sunday January 10, 2016 @11:02PM (#10878709)
That makes sense; the most recent editions of POSIX "import" ISO C '99. I had a copy of POSIX handy to check it, but not ISO C :)
Re:dd built-in progress introduced in coreutils 8.24 by __david__ on Sunday January 10, 2016 @09:58PM (#10878502)
Yeah, it's ugly, but it's sadly the only way to correctly do cross platform prints where the type may or may not be 64 bit.
Re:dd built-in progress introduced in coreutils 8.24 by _kst_ on Sunday January 10, 2016 @10:20PM (#10878576)
The `PRIuMAX` macro in particular is not necessary. It expands to a format usable for `uintmax_t`, but you can also use `"%ju"`. Both were introduced in ISO C99.

(Unless your `printf` implementation doesn't recognize `"%ju"`, and the `PRIuMAX` macro and friends are defined in some non-standard system-specific manner.)

Re:dd built-in progress introduced in coreutils 8.24 by e12e on Monday January 11, 2016 @12:14AM (#10878955)
Related, the recently posted "How to write C in 2016":

https://news.ycombinator.com/item?id=10864176

Re:dd built-in progress introduced in coreutils 8.24 by garethrees on Monday January 11, 2016 @06:24AM (#10880179)
How is this supposed to work? The underscore is a convenience macro for the gettext localization function. But gettext works by looking up its string argument in the localization database, and here the string varies from one platform to another, so how can you be sure that the localization will be found?

I guess the answer is that you have to redo the localization for each platform that has a different expansion of PRIuMAX. But that's horrible: the localization will be identical except for the format specifiers, which will just be copied from the English version of the string. Is there support for automating this in the translation toolchain?

Re:dd built-in progress introduced in coreutils 8.24 by arantius on Monday January 11, 2016 @12:51PM (#10881903)
What is the behavior of gettext() if the message is not found?

(This question is the answer to your question.)

Re:dd built-in progress introduced in coreutils 8.24 by garethrees on Wednesday January 13, 2016 @05:09PM (#10897972)
If the message is not found, gettext returns the argument unchanged, which would be a bug (English text incorrectly appearing instead of localized text). So that's not it.

The answer to my question is that xgettext and gettext have special handling for PRIuMAX and the other formatting macros in inttypes.h. http://www.gnu.org/software/gettext/manual/gettext.html#Prep...

Re:dd built-in progress introduced in coreutils 8.24 by proctor on Sunday January 10, 2016 @08:27PM (#10878230)
i've found that pv is useful for this purpose. however i've never been quite clear on what the best way to use bs with pv is. is it

  dd if=/dev/sda bs=1M | pv > sda.file
or is this better

  dd if=/dev/sda bs=1M | pv | dd of=sda.file bs=1M
Re:dd built-in progress introduced in coreutils 8.24 by rogerbinns on Sunday January 10, 2016 @09:21PM (#10878391)
If all you are doing is the above, then you don't need dd at all. pv will act like dd (but with stdin/out) and uses a sensible buffer size you can override. Your first example becomes:

  pv --buffer-size 1m < /dev/sda > sda.file
http://www.ivarch.com/programs/quickref/pv.shtml

  -B BYTES, --buffer-size BYTES
    Use a transfer buffer size of BYTES bytes. A suffix of "k", "m", "g", or
    "t" can be added to denote kilobytes (*1024), megabytes, and so on. The 
    default buffer size is the block size of the input file's filesystem 
    multiplied by 32 (512kb max), or 400kb if the block size cannot be 
    determined.
pv is great tool and works really well. On Linux you can even point it at other process ids, and it will automatically scan their open file descriptors and show progress of relevant ones.
Re:dd built-in progress introduced in coreutils 8.24 by ckuehl on Monday January 11, 2016 @12:22AM (#10878985)
Just be very careful which direction you point those arrows :-)

I've have similar nightmares about typoing `if` and `of` when using dd.

Re:dd built-in progress introduced in coreutils 8.24 by agumonkey on Monday January 11, 2016 @03:39AM (#10879609)
Quite often, when I'm about to write a command involving `dd`, the first character I type is #.
Re:dd built-in progress introduced in coreutils 8.24 by Tiksi on Monday January 11, 2016 @03:06AM (#10879510)
You can also drop the `<` in that example, pv takes a file to read from as an argument:

  pv --buffer-size 1m /dev/sda > sda.file
Will do the same thing, and give an accurate % complete progress bar.
Re:dd built-in progress introduced in coreutils 8.24 by aktau on Monday January 11, 2016 @04:14AM (#10879725)
> Will do the same thing, and give an accurate % complete progress bar.

I'm not sure that's true, last time I tried to do:

  pv < /dev/zero > /dev/sda
To zero a drive, it correctly printed out the total size of the output block device. Perhaps it also does this type of scanning on stdin, if possible.

That said, looking at the main page, it appears that this functionality is only available for the output end:

Note that if the input size cannot be calculated, and the output is a block device, then the size of the block device will be used and pv will automatically stop at that size as if -S had been given. (http://www.ivarch.com/programs/quickref/pv.shtml)

Re:dd built-in progress introduced in coreutils 8.24 by Tiksi on Monday January 11, 2016 @08:32AM (#10880632)
Ah, I see what you mean, I was thinking in terms of images/files.

As an arguement it gives % of the input (64GB usb drive, 8GB image):

  > pv /mnt/str/Downloads/stick-8g.img > /dev/sdi
  916MiB 0:00:35 [1.88MiB/s] [========>                                                                           ] 11% ETA 0:04:17
Otherwise it seems to give the size of the output if it's a block device.
Re:dd built-in progress introduced in coreutils 8.24 by zx2c4 on Monday January 11, 2016 @06:06AM (#10880116)
There's also this incredible tool that spies on processes to get real time progress:

https://github.com/Xfennec/progress

Re:dd built-in progress introduced in coreutils 8.24 by zimbatm on Sunday January 10, 2016 @07:49PM (#10878084)
dd was always a big special to me. It's the only command-line utility that I know that takes the simple approach to argument parsing. I don't know why everyone wants to prefix their arguments with --
Re:dd built-in progress introduced in coreutils 8.24 by pixelbeat on Sunday January 10, 2016 @07:53PM (#10878114)
As mentioned in the online manual at http://gnu.org/software/coreutils/dd that command line syntax is inspired by the DD (data definition) statement of OS/360 JCL
Re:dd built-in progress introduced in coreutils 8.24 by kazinator on Monday January 11, 2016 @02:49AM (#10879447)
For one thing, nobody wants to type:

  mv opt=force old=oldname.txt new=/path/to/new.txt
In dd it doesn't matter because it's used for "surgical" jobs where you have to think very carefully about several simultaneous numeric details.
Re:dd built-in progress introduced in coreutils 8.24 by phjesusthatguy3 on Sunday January 10, 2016 @07:54PM (#10878126)
I've been using dcfldd for years now, just because of the progress bar.
Re:dd built-in progress introduced in coreutils 8.24 by relaxitup on Sunday January 10, 2016 @11:41PM (#10878845)
ddrescue also shows progress by default. Not sure if it has a comparable feature-set as dcfldd, but for data rescue/forensics, it definitely does the trick, and seems to be much more actively maintained.
Re:dd built-in progress introduced in coreutils 8.24 by vdfs on Sunday January 10, 2016 @06:31PM (#10877761)
Example (from Reddit post): # dd if=arch.iso of=/dev/sdb bs=4M status=progress 61432564 bytes (61 MB) copied, 3.024017 s, 20.3 MB/s
Re:dd built-in progress introduced in coreutils 8.24 by iofj on Sunday January 10, 2016 @07:19PM (#10877941)
Is a progress bar or percentage really that much to ask ?
Re:dd built-in progress introduced in coreutils 8.24 by amalcon on Sunday January 10, 2016 @08:27PM (#10878231)
You don't always know how many bytes remain to be copied, e.g. in the case of a FIFO. It could theoretically do things differently in the cases where you do know, however.
Re:dd built-in progress introduced in coreutils 8.24 by gedsic on Monday January 11, 2016 @03:59AM (#10879672)
I get "No repositories found"...
Re:dd built-in progress introduced in coreutils 8.24 by rkangel on Monday January 11, 2016 @06:57AM (#10880276)
I get "no repositories found" for that link.
Re:dd built-in progress introduced in coreutils 8.24 by rjeli on Monday January 11, 2016 @02:24AM (#10879350)
See: cat -v Considered Harmful by Rob Pike
Re:dd built-in progress introduced in coreutils 8.24 by codemac on Monday January 11, 2016 @02:53AM (#10879458)
This is a pretty useful feature for something that regularly copies large files/disks you'd want to see the progress of, in fact I'd say it should have been default behavior for years.

I'd make some crude comment about cargo culting, but do you have an explicit complaint? dd itself is derided by the plan9 crew for many more reasons than a progress bar, and the cli arguments are obtuse because they're a joke.

[0]: http://www.catb.org/jargon/html/D/dd.html

Re:dd built-in progress introduced in coreutils 8.24 by mahouse on Sunday January 10, 2016 @06:41PM (#10877797)
Couldn't you just send a SIGUSR1 signal to see the progress?
Re:dd built-in progress introduced in coreutils 8.24 by pixelbeat on Sunday January 10, 2016 @07:56PM (#10878131)
That's more awkward and outputs a separate line per interrupt. Also to do that programmatically (to provide progress to a program using dd underneath) is tricky to do robustly due to the default disposition of SIGUSR1. SIGUSR1 is really only a hack provided on systems that don't support SIGINFO (which is easier to generate and whose default disposition is easier to handle robustly)
Re:dd built-in progress introduced in coreutils 8.24 by mshook on Sunday January 10, 2016 @06:55PM (#10877859)
You could but letting dd take care of it instead of doing it yourself is much better IMHO.
Re:dd built-in progress introduced in coreutils 8.24 by zimbatm on Sunday January 10, 2016 @07:46PM (#10878069)
What if you started dd without status=progress ?
Re:dd built-in progress introduced in coreutils 8.24 by Gracana on Sunday January 10, 2016 @10:21PM (#10878586)
Updating dd to continuously dump text to the screen by default would probably break a lot of existing scripts.
Re:dd built-in progress introduced in coreutils 8.24 by zimbatm on Monday January 11, 2016 @04:53AM (#10879860)
Just on SIGUSR1 would be good enough to get an idea where dd is at. And it would be on stderr, not stdout.
Re:dd built-in progress introduced in coreutils 8.24 by Gracana on Monday January 11, 2016 @10:16AM (#10881181)
> it would be on stderr, not stdout

Oh, that's true.

Anyway, what behavior do you want to see exactly? You can still send the USR1 signal and get the update behavior just like before. Were you just pointing out that there are cases where you still need to use that feature, or something else?

Re:dd built-in progress introduced in coreutils 8.24 by neikos on Sunday January 10, 2016 @09:08PM (#10878353)
Then that's your own fault?
Re:dd built-in progress introduced in coreutils 8.24 by zimbatm on Monday January 11, 2016 @04:52AM (#10879858)
But surely if you could send a SIGUSR1 and get the progress it would be useful. I won't always have the foresight to put the progress argument. Maybe I expected the task to be quick but it's not.
Re:dd built-in progress introduced in coreutils 8.24 by unluckier on Monday January 11, 2016 @01:11AM (#10879133)
If you're on BSD, that'll kill dd.