News for hackers. Stuff that matters.
Re:Dd is not a disk writing tool (2015) by voidz on Sunday January 08, 2017 @07:40AM (#13349475)
Most of the time it's much better (as in, faster) to just use cat (or pv, to get a nice progress bar) for writing a file to a block device, because it streams, and lets underlying heuristics worry about block sizes and whatnot.

So:

  cat foobar.img > /dev/sdi
will stream the file rather than what dd does, i.e. read block, write block, read block, write block and so on.

Usually I also lower the vm.dirty_bytes and vm.dirty_background_bytes to 16 resp. 48 MB (in bytes) beforehand, which limits the buffer sizes to those amounts. Else it will seem that the progress bar indicates 300MB/s is written, and when it completes you still have to wait a really long time for things to have been written out. Afterwards I restore back vm.dirty_ratio and vm.dirty_background_ratio to respectively 10 and 5 - the defaults on my system.

I wish that all of those projects, tutorials etc. that explain how to write their image to a block device, like an sdcard, would start advise using cat, because there is no reason to use dd, it's just something that people stick with because others do it too.

I only use dd for specific blocks, like writing back a backup of the mbr, or as a rudimentary hex editor.

Re:Dd is not a disk writing tool (2015) by fragmede on Sunday January 08, 2017 @12:15PM (#13350727)
> So:

> cat foobar.img > /dev/sdi

> will stream the file rather than what dd does,

Sorry, that's not quite right. `cat` (and your shell, presumably bash) does the same fundamental thing as `dd`, ie, read block, write block. There's not really an underlying 'stream' primitive that `cat` (or `bash`, as you're using redirection to write the file) is using compared to `dd`

What `cat` does do, is it does a better job of trying to find an optimal blocksize than a naive `dd` call does. `dd` simply defaults to a 512-byte block size, which is just inherited from history when 512-bytes was the alignment for, well, everything.

There are numerous optimizations upon the fundamental read-block, write-block primitives to make it go faster (`cat` makes use of some of these). The linux kernel actually has a "stream file to socket" syscall to avoid the copy to user-land and back to "stream" a file out to the network, but that's not happening here, and there's still reading and writing of blocks in the kernel happening.

See also: http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;f...

Re:Dd is not a disk writing tool (2015) by LukeShu on Sunday January 08, 2017 @01:40PM (#13351179)
That's still not quite right.

Once cat is spawned, the shell gets out of the way, and has nothing to do with it. (Exception: zsh will "helpfully" insert itself with `tee`-like operation in certain situations.)

What `cat` doesn't do is finding an optimal block size for writes. It does a good job of finding optimal block size for reads, but not for writes. Many older block devices perform very poorly when the write block size is not optimal.

Re:Dd is not a disk writing tool (2015) by omribahumi on Sunday January 08, 2017 @04:09PM (#13352029)
The fact that zsh is a parent doesn't necessarily hurt performance. The parent and child can share the stdin fd, and the child could be the only one reading from it.
Re:Dd is not a disk writing tool (2015) by LukeShu on Sunday January 08, 2017 @06:56PM (#13352921)
That's not what I was referring to. Though I should have been more clear: I don't believe that zsh inserts itself in the mentioned situation; just that it does insert itself in some situations, and that makes it an exception to my general "once the process is spawned, the calling shell doesn't matter" statement.

As an example of what I was referring to, in zsh, this

    a >&2 | b
is equivalent to Bourne shell:

    a | tee /dev/stderr | b
except that `tee` is implemented as part of zsh, rather than a separate program (for no difference in performance).

That is, zsh inserted itself into the middle of the pipeline. There are two pipes instead of one; zsh/tee reads from the a pipe, and then writes that data to the b pipe (and stderr). This does hurt performance.

Re:Dd is not a disk writing tool (2015) by asdfaoeu on Sunday January 08, 2017 @09:31PM (#13353539)
This doesn't seem to be the case for me

    -> % python test.py 2>&1 | python test.py
    COMMAND   PID    USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
    [snip]
    python  24150 someone    0u   CHR  136,0      0t0        3 /dev/pts/0
    python  24150 someone    1w  FIFO    0,8      0t0 72772851 pipe
    python  24150 someone    2w  FIFO    0,8      0t0 72772851 pipe
    python  24150 someone    3w   CHR    5,0      0t0     1049 /dev/tty
    COMMAND   PID    USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
    [snip]
    python  24151 someone    0r  FIFO    0,8      0t0 72772851 pipe
    python  24151 someone    1u   CHR  136,0      0t0        3 /dev/pts/0
    python  24151 someone    2u   CHR  136,0      0t0        3 /dev/pts/0
    python  24151 someone    3w   CHR    5,0      0t0     1049 /dev/tty
    someone@arch-two [01:29:00] [~]
    -> % echo $ZSH_VERSION
    5.2
    someone@arch-two [01:29:16] [~]
    -> % cat test.py
    import os
    import sys
    import subprocess

    with open("/dev/tty", "w") as f:
        subprocess.check_call(["lsof", "-p", str(os.getpid())], stdout=f)

Edit: Ok so I misread your comment and you were talking about >&2. Which is weird because it actually has different behaviour in bash vs zsh. Having said I can't think why you would want the bash behaviour, although the bash behaviour does seem more consistent.
Re:Dd is not a disk writing tool (2015) by LukeShu on Sunday January 08, 2017 @10:09PM (#13353707)
In the particular example I used, the Bourne shell behavior probably has no practical use (but I can write code of no practical use in many languages).

It's not particularly surprising that the zsh behavior is different than bash; zsh redirections are only Bourne shell-like in simple cases.

I dislike zsh's behavior because it isn't what the user typed. It should be simple to look at a command line and see how many pipes it makes; see the processes that will be involved in the pipeline. Zsh's behavior seems to me to be clever and implicit; I'd much rather have dumb and explicit.

Re:Dd is not a disk writing tool (2015) by RJIb8RBYxzAMX9u on Sunday January 08, 2017 @09:11AM (#13349727)
> I wish that all of those projects, tutorials etc. that explain how to write their image to a block device, like an sdcard, would start advise using cat, because there is no reason to use dd, it's just something that people stick with because others do it too.

I'd wondered whether dd or cat were faster, and indeed cat is faster, but not by much. Also, for some embedded devices, you have to write to specific offsets, so dd is more convenient and explicit. Lastly, cat composes poorly with sudo.

  $ sudo dd if=foobar.img of=/dev/sdi # works
  $ sudo cat foobar.img > /dev/sdi    # fails unless root b/c redirection is done by shell
> I only use dd for specific blocks, like writing back a backup of the mbr, or as a rudimentary hex editor.

xxd / xxd -r is much nicer, but I suppose sometimes vim is not available...

Re:Dd is not a disk writing tool (2015) by JoshTriplett on Sunday January 08, 2017 @10:20AM (#13350002)
> I'd wondered whether dd or cat were faster, and indeed cat is faster, but not by much.

That performance difference often comes from block size; "dd bs=1M" typically runs much faster than the default block size of 512 bytes.

Re:Dd is not a disk writing tool (2015) by mercora on Sunday January 08, 2017 @10:19AM (#13349994)
one could use:

cat foobar.img | sudo tee /dev/sdi > /dev/null

Re:Dd is not a disk writing tool (2015) by JoshTriplett on Sunday January 08, 2017 @10:23AM (#13350014)
I've used that to write short files (such as settings in /sys or /proc), but for large files, tee has the disadvantage of writing everything twice, and the pipe adds another write and read of every byte.
Re:Dd is not a disk writing tool (2015) by mercora on Sunday January 08, 2017 @10:28AM (#13350048)
I just realized this too and tried to close stdout instead. tee complains about it but goes on with its business:

cat foobar.img | sudo tee /dev/sdi >&-

Re:Dd is not a disk writing tool (2015) by RJIb8RBYxzAMX9u on Sunday January 08, 2017 @10:40AM (#13350126)
Why's everyone so against dd? :-) If you're going to use tee, might as well not bother with cat at all[0]:

  $ sudo tee /dev/sdi >&- < foobar.img
Or better yet, pv[1]; you even get a progress bar that way!

[0] https://en.wikipedia.org/wiki/Cat_%28Unix%29#Useless_use_of_...

[1] http://www.ivarch.com/programs/pv.shtml

Re:Dd is not a disk writing tool (2015) by mercora on Sunday January 08, 2017 @10:59AM (#13350249)
Thanks for the reminder :) I do fall quite often for the useless use of cat. But most of the time i also do not really care about it much. I did omit pv in believe tee will always be available but it is great and absolutely preferred when available.
Re:Dd is not a disk writing tool (2015) by carussell on Sunday January 08, 2017 @10:57AM (#13350240)
> xxd / xxd -r is much nicer, but I suppose sometimes vim is not available

`od` is ubiquitous--it's POSIX and a requirement of the Single UNIX Specification

Re:Dd is not a disk writing tool (2015) by nerdponx on Sunday January 08, 2017 @09:27AM (#13349785)
Would

    sudo (cat foobar.img > /dev/sdi)
Work?
Re:Dd is not a disk writing tool (2015) by sply on Sunday January 08, 2017 @09:57AM (#13349887)
No, you should use

  sudo sh -c "cat foobar.img > /dev/sdi"
or

  sudo -s "cat foobar.img > /dev/sdi"
Re:Dd is not a disk writing tool (2015) by RJIb8RBYxzAMX9u on Sunday January 08, 2017 @10:29AM (#13350061)
I was originally going to put that in my examples, but opted to leave it out, because with sh -c you have to think about escaping special characters. Most of the time it doesn't matter, but when running commands as root you ought to be absolutely sure.
Re:Dd is not a disk writing tool (2015) by nerdponx on Sunday January 08, 2017 @09:41PM (#13353586)
What about using a here-doc (as in http://stackoverflow.com/a/16514624/2954547)?

    sudo sh <<EOF
    cat foobar.img > /dev/sdi
    EOF
Re:Dd is not a disk writing tool (2015) by RJIb8RBYxzAMX9u on Sunday January 08, 2017 @09:36AM (#13349814)
No, unless you have a shell where sudo is a built-in, and I don't know of any.
Re:Dd is not a disk writing tool (2015) by c0l0 on Sunday January 08, 2017 @10:03AM (#13349915)
This is actually impossible (at least with UNIX shells), as shell builtins cannot have the SUID bit set - and your shellitself also shouldn't ;)
Re:Dd is not a disk writing tool (2015) by JoshTriplett on Sunday January 08, 2017 @10:19AM (#13349990)
You could have a shell with a "sudo" builtin that knew how to invoke a separate "sudo" program with the right shell syntax and quoting, such that "sudo somecommand > /path/to/root-writable-file" did the right thing.
Re:Dd is not a disk writing tool (2015) by abofh on Sunday January 08, 2017 @12:29PM (#13350809)
Not really - to sudo, your shell would have to be setuid - and constantly fork stuff as you to get user permissions. Alternatively your shell could maintain a separate process for privileged access, but that puts a whole lot of your security on the assumption that your shell has no bugs that might allow escalation.

In short, you could do it, but it'd be ripped out of every server that's been hardened, and for users that don't want to care - they're just running 'sudo su' anyhow.

Speaking only for myself, the thought of my shell having a magical escalation process would scare the bejeezus out of me - and I'm supposed to have root on our boxes!

Re:Dd is not a disk writing tool (2015) by TheCoelacanth on Sunday January 08, 2017 @03:42PM (#13351875)
The shell wouldn't need to be setuid if it just performed a syntax transformation and then called the existing sudo binary.
Re:Dd is not a disk writing tool (2015) by JoshTriplett on Sunday January 08, 2017 @06:24PM (#13352741)
Exactly. The shell would just treat "sudo" as a builtin prefix similar to "time", but would then run the real "sudo" with an appropriate shell invocation and proper quoting.
Re:Dd is not a disk writing tool (2015) by nerdponx on Sunday January 08, 2017 @09:44PM (#13353604)
I finally got home to try it, and indeed it does fail (Zsh).

    zsh: parse error near `)'
Is it just ignoring the parentheses? Does the redirection "take precedence" over parentheses and brackets?
Re:Dd is not a disk writing tool (2015) by na85 on Sunday January 08, 2017 @11:30AM (#13350459)
Sudo is anachronistic anyways.

Laptops don't need to follow multiuser best practices, and frankly a complex root password offers little.

Re:Dd is not a disk writing tool (2015) by greggyb on Sunday January 08, 2017 @09:26AM (#13349781)

    sudo (cat foobar.img > /dev/sdi)
No?
Re:Dd is not a disk writing tool (2015) by ryao on Sunday January 08, 2017 @11:35AM (#13350492)
That is a syntax error in bash.
Re:Dd is not a disk writing tool (2015) by greggyb on Monday January 09, 2017 @02:58PM (#13359287)
Learn something new every day. Thanks.
Re:Dd is not a disk writing tool (2015) by mbrumlow on Sunday January 08, 2017 @12:22PM (#13350773)
This is all wrong.

I am not even sure what you mean by stream or how it would be any difference than "read", "write", "read", "write". Because that is literally what cat does, and so does dd.

Many people feel cat is faster than dd because dd's default block size is 512 bytes as you can see from a simple dd command.

  strace dd if=10M of=junk 

  read(0, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
  write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
  read(0, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
  write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
  read(0, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
  write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512

  strace cat 10M > junk2

  read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 131072) = 131072
  write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 131072) = 131072
  read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 131072) = 131072
  write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 131072) = 131072
  read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 131072) = 131072
  write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 131072) = 131072
So when benchmarking dd vs cat make sure you are comparing apples to apples.

The beauty of dd is you don't have to tinker with things like vm-.dirty_bytes and vm.dirty_background_bytes before using a command. Those should not be messed with on most systems and should definitely not be messed with for the sake of running a single command. When you use cat, it makes some decisions for you. Most of the time makes good choices but it happens to make not so optimal choices for working with large files being written to other devices or file systems.

To avoid the pitfalls of using cat to copy image files (and not have to change global system settings) you use dd. With dd you can use O_DIRECT to bypass the VFS caching layer so your vm.dirty_* settings are ignored in general. You can also specify the block size optimal for the device you are writing to and even reading from.

While dd may not stand for disk or drive or its name have anything to do with disk it is a powerful tool that is the appropriate tool to use when working with large files, precise movement or data and yes copying image files to devices.

Re:Dd is not a disk writing tool (2015) by quotemstr on Sunday January 08, 2017 @04:19PM (#13352078)
Almost, but not quite. You really want buffer in there to smooth out IO time variances.

    <foo.img pv -cNsrc | buffer -p75 -s256k -b128 | pv -cNdst >/dev/whatever
buffer lets the rest of the pipeline proceed while pv is blocked writing to the output file. Yes, conventional filesystem readahead helps to some extent, but IME, not enough, especially when foo.img is a block device itself.
Re:Dd is not a disk writing tool (2015) by wpietri on Sunday January 08, 2017 @04:27PM (#13352122)
Holy moly. I have spent a lot of years typing Unix commands, but it never occurred to me to put the input redirection first. But that's much more pipeline-ish, so I like it a lot.