News for hackers. Stuff that matters.
Re:Dd is not a disk writing tool (2015) by pixelbeat on Sunday January 08, 2017 @06:58AM (#13349360)
Yes "D" is not for disk/drive/device/...

It comes from the DD (data definition) statement of OS/360 JCL, and hence why dd has the unusual option syntax compared to other unix utils

BTW if you are using dd to write usb drives etc. it's useful to bypass the Linux VM as much as possible to avoid systems stalls, especially with slow devices. You can do that with O_DIRECT. Also dd recently got a progress option, so...

    dd bs=2M if=disk.img of=/dev/sda... status=progress iflag=direct oflag=direct
Note dd is a lower level tool, which is why there are some gotchas when using for higher level operations. I've noted a few at:

http://www.pixelbeat.org/docs/coreutils-gotchas.html#dd

Re:Dd is not a disk writing tool (2015) by carussell on Sunday January 08, 2017 @11:34AM (#13350487)
You don't have to wait for updates to newer versions to get dd to report its progress. The status line that dd prints when it finishes can also be forced at any point during dd's operation by sending the USR1 or INFO signals to the process. E.g.:

    ps a | grep "\<dd"
    # [...]
    kill -USR1 $YOUR_DD_PID
or

    pkill -USR1 ^dd
It also doesn't require you to get everything nailed down at the beginning. You've just spent the last 20 seconds waiting and realize you want a status update, but you didn't think to specify the option ahead of time? No problem.

I've thought that dd's behavior could serve as a model for a new standard of interaction. Persistent progress indicators are known to cause performance degradation unless implemented carefully. And reality is, you generally don't need something to constantly report its progress even while you're not looking, anyway.

To figure out the ideal interaction, try modeling it after the conversation you'd have if you were talking to a person instead of your shell:

"Hey, how much longer is it going to take to flash that image?"

The way dd works is close to this scenario.

Re:Dd is not a disk writing tool (2015) by pixelbeat on Sunday January 08, 2017 @11:42AM (#13350536)
Yes this is true. Note BSD supports this better with Ctrl-T to generate SIGINFO, which one can send to any command even if not supported, in which case it's ignored. Using kill on linux, and having that kill processes by default is decidedly more awkward.

It's also worth noting the separate "progress" project which can be used to give the progress of running file based utilities.

We generally have pushed back on adding progress to each of the coreutils for these reasons, but the low overhead of implementation and high overlap with existing options was deemed enough to warrant adding this to dd

Re:Dd is not a disk writing tool (2015) by gkya on Sunday January 08, 2017 @04:49PM (#13352242)
Also a gotcha on BSD (FBSD at least): SIGUSR1 kills dd.
Re:Dd is not a disk writing tool (2015) by stuaxo on Sunday January 08, 2017 @05:22PM (#13352412)
Ctrl-T for SIGINFO is pretty useful, it would be good if Linux could pick this up.
Re:Dd is not a disk writing tool (2015) by circuit_breaker on Sunday January 08, 2017 @06:05PM (#13352629)
Been waiting for years, frankly. I guess if I was motivated enough and had the time, I could do the research and submit patches..
Re:Dd is not a disk writing tool (2015) by feld on Sunday January 08, 2017 @07:53PM (#13353181)
They don't want it. Even if you got the patches accepted to the kernel they'd never accept patches to GNU Coreutils to support it.
Re:Dd is not a disk writing tool (2015) by feld on Monday January 09, 2017 @03:04PM (#13359345)
The most recent comment I've seen on this is lukewarm at best: http://lkml.iu.edu/hypermail/linux/kernel/1411.0/03374.html
Re:Dd is not a disk writing tool (2015) by floatboth on Monday January 09, 2017 @07:28AM (#13355826)
> in which case it's ignored

Ignored by the application, sure. But FreeBSD always prints useful stuff like load, current command, its pid and state:

$ dd if=/dev/random of=/dev/null

load: 0.72 cmd: dd 5820 [running] 0.70r 0.02u 0.68s 6% 2008k

263276+0 records in

263276+0 records out

134797312 bytes transferred in 0.708372 secs (190291809 bytes/sec)

(here, the "load:..." line is from the system, and the other 3 lines are from dd)

Re:Dd is not a disk writing tool (2015) by highd on Sunday January 08, 2017 @01:43PM (#13351195)
Yes! I'm thinking of building something like this for my neural net training (1-2 days on AWS, 16 GPUs/processes on the job). In this case the "state" that I'd like to access is all the parameters of the model and training history, so I'm thinking I'll probably store an mmapped file so I can use other processes to poke at it while it's running. That way I can decouple the write-test-debug loops for the training code and the viz code.
Re:Dd is not a disk writing tool (2015) by dllthomas on Sunday January 08, 2017 @06:06PM (#13352637)
> I'm thinking I'll probably store an mmapped file so I can use other processes to poke at it while it's running.

That seems to run substantial risk of seeing it in an inconsistent state, yeah?

Re:Dd is not a disk writing tool (2015) by highd on Monday January 09, 2017 @12:28AM (#13354226)
I generally use a semaphore when I'm reading and writing from my shm'd things. The data structure will also likely be append-only for the training process, as I want to see how things are changing over time.

Also I meant shm'd, not mmap'd.

Re:Dd is not a disk writing tool (2015) by terminalcommand on Monday January 09, 2017 @02:05PM (#13358828)
I am knew to the shared memory concept. I am familiar with named pipes. Could you please elaborate a bit, I'm curios.

Are you passing the reference to an mmap adress, or using the shm systemcalls? In what language are you programming in? Does race conditions endanger the shared memory? If so, how does using semaphores help?

Sorry, if I asked a lot of questions, feel free to answer any/none of them :)

Re:Dd is not a disk writing tool (2015) by highd on Monday January 09, 2017 @09:38PM (#13362081)
Sure! SHM is really cool, I just found out about it. It's an old Posix functionality, so people should use it more!

I'm using shm system calls in Python. Basically I get a buffer of raw bytes of a fixed size that is referred to by a key. When I have multiple processes running I just have to pass that key between them and they get access to that buffer of bytes to read and write.

On each iteration first I wait until the semaphore is free and then I lock it (P). That prevents anyone else from accessing the shared memory. I have the process read from the shared memory a set of variables - I have little helper functions that serialize and deserialize numpy arrays into raw bytes using fixed shapes and dtypes. Those arrays are then updated using some function combining the output of the process and the current value of the array. Then those arrays are reserialized and written back to the shm buffer as raw bytes again. Finally, the process releases the semaphore using P() so other processes can access it. The purpose of the semaphore is to prevent reading the arrays while another process is writing them - otherwise you might get interleaved old and new data from a given update. In a process-wise sense there is a race-condition, as each process can update at different times or in a different order, but for my purposes this is acceptable since neural net training is a stochastic sort of thing and it shouldn't care too much.

[0] http://nikitathespider.com/python/shm/ - original library which works fine for me

[1] http://semanchuk.com/philip/PythonIpc/ - updated version

Re:Dd is not a disk writing tool (2015) by hueving on Sunday January 08, 2017 @11:39AM (#13350519)
>I've thought that dd's behavior could serve as a model for a new standard of interaction. Persistent progress indicators are known to cause performance degradation unless implemented carefully. And reality is, you generally don't need something to constantly report its progress even while you're not looking, anyway.

Progress bars by default are also garbage if you are scripting and want to just log results. ffmpeg is terrible for this.

Re:Dd is not a disk writing tool (2015) by jff on Sunday January 08, 2017 @12:12PM (#13350709)
> Persistent progress indicators are known to cause performance degradation unless implemented carefully.

Are you referring to that npm progress bar thing a few months back? I'm pretty sure the reason for that can be summed up as "javascript, and web developers".

Anyway, he's not proposing progress bars by default, he's proposing a method by which you can query a process to see how far it's come. I think there's even a key combination to do this on FreeBSD.

Or, for example, you could write a small program that sends a USR1 signal every 5 seconds, splitting out the responsibility of managing a progress bar:

% progress cp bigfile /tmp/

And then the 'progress' program would draw you a text progress bar, or even pop up an X window with a progress bar.

Re:Dd is not a disk writing tool (2015) by hashhar on Monday January 09, 2017 @02:36AM (#13354679)
Re:Dd is not a disk writing tool (2015) by jff on Monday January 09, 2017 @11:15AM (#13357393)
That's great! I think due to the way it's implemented it wouldn't be able to do progress reporting for e.g. "dd if=/dev/zero of=bigfile bs=1M count=2048", but that's a less common case than just cp'ing a big "regular" file.
Re:Dd is not a disk writing tool (2015) by erikbye on Sunday January 08, 2017 @01:04PM (#13351000)
Yes, C-t for SIGINFO, works on all BSDs (including macOS).
Re:Dd is not a disk writing tool (2015) by joombaga on Monday January 09, 2017 @12:29AM (#13354229)
I've always used pv to get progress from dd, or other pipes:

  pv image.img | dd of=/dev/rdisk2 bs=1M
This adds another pipe though. I don't know the effect this has on performance.
Re:Dd is not a disk writing tool (2015) by lloeki on Sunday January 08, 2017 @07:33AM (#13349454)
On OS X (and BSD?) be sure to use /dev/rdisk[0-9]+ instead of /dev/disk[0-9]+.

Details as to exactly why it's faster are welcome. (I just know it bypasses stuff).

EDIT: someone mentioned this below http://superuser.com/questions/631592/why-is-dev-rdisk-about...

Re:Dd is not a disk writing tool (2015) by mkup on Sunday January 08, 2017 @08:43AM (#13349658)
In FreeBSD, cached/block disk devices are long gone: https://www.freebsd.org/doc/en/books/arch-handbook/driverbas... so all disk devices in /dev are implicitly O_DIRECT.

Though, read cache can be enabled manually by creating separate device via gcache(8). This is usually not required, because caching is done at the filesystem layer.

It's important to specify block size for uncached devices, of course. dd(1) with bs= option will surely work, and with cp(1) your mileage may vary, depending on whether underlying disk driver supports I/O with partial sector size or not.

Re:Dd is not a disk writing tool (2015) by djent on Sunday January 08, 2017 @08:34AM (#13349628)
Ah wish I knew this last week. Writing Xubuntu to my USB took something like 2900 seconds from a Mac.
Re:Dd is not a disk writing tool (2015) by josteink on Sunday January 08, 2017 @11:56AM (#13350598)
Usually just specifying a reasonable blocksize works for me. bs=1m or so.

Without that it does literally take hours.

I suspect the default blocksize is really small (1?) and combined with uncached/unbuffered writes to slower devices, it just kills all performance outright.

Edit: answered! https://news.ycombinator.com/item?id=13350002

Re:Dd is not a disk writing tool (2015) by NamTaf on Sunday January 08, 2017 @07:45PM (#13353148)
Per the sibling comments, you just need to specify a sane block size. dd's default is really low and if you experiment a bit with 2M or around that you'll get near-theoretical throughput.

NB: Remember the units! Without the units you specify it as bytes or something insanely small like that. I've made that mistake more than once!

Re:Dd is not a disk writing tool (2015) by muterad_murilax on Sunday January 08, 2017 @09:02AM (#13349691)
In other words, about 48 minutes for a ~1.2 GB file?
Re:Dd is not a disk writing tool (2015) by isostatic on Sunday January 08, 2017 @11:46AM (#13350546)
About 3mbit/second, or 400kbytes a second. I'd expect something 50-100 times faster.
Re:Dd is not a disk writing tool (2015) by rsync on Sunday January 08, 2017 @02:18PM (#13351383)
"Yes "D" is not for disk/drive/device/..."

But that's the very beauty of unix!

If you can find a way to use 'dd' for disk/drive/device you can use it in interesting new manners (pipelines, etc.) and have very good confidence that it won't break in weird ways. It will do the small, simple thing it is supposed to do even if you are abusing it horribly.

Like this, for instance:

  pg_dump -U postgres db | ssh [email protected] "dd of=db_dump"
Re:Dd is not a disk writing tool (2015) by takeda on Sunday January 08, 2017 @03:50PM (#13351933)
Is there a benefit to use dd over cat in this case?
Re:Dd is not a disk writing tool (2015) by circuit_breaker on Sunday January 08, 2017 @06:43PM (#13352832)
You could use it to rate limit... or arbitrarily set block sizes per use case. I've used it for the former when doing 'over the wire' backups through ssh
Re:Dd is not a disk writing tool (2015) by abrowne on Sunday January 08, 2017 @02:44PM (#13351530)
Thanks for the tips.

Clueless noob here . . . most guides I've seen use bs=1M for writing e.g. a Linux installer to a USB drive. Does 1 MB vs 2 MB change anything?

Re:Dd is not a disk writing tool (2015) by blr246 on Sunday January 08, 2017 @04:30PM (#13352128)
The setting controls the block size. When writing to block devices, you can maximize throughput by tuning the block size for the filesystem, architecture, and specific disk drive in use. You can tune it by benchmarks and searching over various multiples of 512K block sizes.

For most modern systems, 1MB is a reasonable place to start. Even as high as 4MB can work well.

The block size can make a major difference in terms of sustained write speed due to reduced overhead in system calls and saturation of the disk interface.

A similar thing happens when writing to sockets where lots of small messages kill throughput, but they can decrease latency for a system that passes a high volume of small control messages.

Re:Dd is not a disk writing tool (2015) by SSLy on Sunday January 08, 2017 @08:30AM (#13349616)
>it's useful to bypass the Linux VM as much as possible to avoid systems stalls

Oh man, I didn't even know that was the cause of these problems.

Re:Dd is not a disk writing tool (2015) by drvdevd on Sunday January 08, 2017 @09:25AM (#13349778)
Nice! Thanks. Built-in status and that Linux bypass trick are beautiful.
Re:Dd is not a disk writing tool (2015) by Vosporos on Sunday January 08, 2017 @10:42AM (#13350144)
Oh? I had read that its name was originally "copy and convert" but `cc` was already taken by the compiler
Re:Dd is not a disk writing tool (2015) by realfinkployd on Sunday January 08, 2017 @01:54PM (#13351264)
Yeah, this article is wrong. Have you ever noticed the syntax for DD is unusual? It is set up more like a JCL syntax.
Re:Dd is not a disk writing tool (2015) by the_mitsuhiko on Sunday January 08, 2017 @07:25AM (#13349427)
I thought it stands for copy and convert but cc was taken bu the c compiler.
Re:Dd is not a disk writing tool (2015) by gonzo on Sunday January 08, 2017 @07:32AM (#13349450)
Ask a graybeard.
Re:Dd is not a disk writing tool (2015) by pmorici on Sunday January 08, 2017 @11:13AM (#13350361)
Hasn't the ability to check progress be around forever? You could always send it SIGUSR1 and get back a progress report on stderr.
Re:Dd is not a disk writing tool (2015) by _joel on Sunday January 08, 2017 @11:20AM (#13350402)
Sure it's been around for a while, GNU version on linux at least. Personally found pipe viewer (pv) quite handy too https://www.ivarch.com/programs/pv.shtml - available in most distros
Re:Dd is not a disk writing tool (2015) by groovy2shoes on Monday January 09, 2017 @04:21AM (#13355095)
Yup. Just don't do what I did earlier and `pkill -USR1 -f dd` if your desktop session is currently being provided to you courtesy of `sddm` …

As a programmer, it usually pays off to be on the lazy side, but every once in a while it comes back and bites me in the arse ;)

Re:Dd is not a disk writing tool (2015) by Esau on Sunday January 08, 2017 @08:12AM (#13349561)
Thank you for the historical information.