> cat foobar.img > /dev/sdi
> will stream the file rather than what dd does,
Sorry, that's not quite right. `cat` (and your shell, presumably bash) does the same fundamental thing as `dd`, ie, read block, write block. There's not really an underlying 'stream' primitive that `cat` (or `bash`, as you're using redirection to write the file) is using compared to `dd`
What `cat` does do, is it does a better job of trying to find an optimal blocksize than a naive `dd` call does. `dd` simply defaults to a 512-byte block size, which is just inherited from history when 512-bytes was the alignment for, well, everything.
There are numerous optimizations upon the fundamental read-block, write-block primitives to make it go faster (`cat` makes use of some of these). The linux kernel actually has a "stream file to socket" syscall to avoid the copy to user-land and back to "stream" a file out to the network, but that's not happening here, and there's still reading and writing of blocks in the kernel happening.
See also: http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;f...
Once cat is spawned, the shell gets out of the way, and has nothing to do with it. (Exception: zsh will "helpfully" insert itself with `tee`-like operation in certain situations.)
What `cat` doesn't do is finding an optimal block size for writes. It does a good job of finding optimal block size for reads, but not for writes. Many older block devices perform very poorly when the write block size is not optimal.
As an example of what I was referring to, in zsh, this
a >&2 | b
is equivalent to Bourne shell: a | tee /dev/stderr | b
except that `tee` is implemented as part of zsh, rather than a separate program (for no difference in performance).That is, zsh inserted itself into the middle of the pipeline. There are two pipes instead of one; zsh/tee reads from the a pipe, and then writes that data to the b pipe (and stderr). This does hurt performance.
-> % python test.py 2>&1 | python test.py
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
[snip]
python 24150 someone 0u CHR 136,0 0t0 3 /dev/pts/0
python 24150 someone 1w FIFO 0,8 0t0 72772851 pipe
python 24150 someone 2w FIFO 0,8 0t0 72772851 pipe
python 24150 someone 3w CHR 5,0 0t0 1049 /dev/tty
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
[snip]
python 24151 someone 0r FIFO 0,8 0t0 72772851 pipe
python 24151 someone 1u CHR 136,0 0t0 3 /dev/pts/0
python 24151 someone 2u CHR 136,0 0t0 3 /dev/pts/0
python 24151 someone 3w CHR 5,0 0t0 1049 /dev/tty
someone@arch-two [01:29:00] [~]
-> % echo $ZSH_VERSION
5.2
someone@arch-two [01:29:16] [~]
-> % cat test.py
import os
import sys
import subprocess
with open("/dev/tty", "w") as f:
subprocess.check_call(["lsof", "-p", str(os.getpid())], stdout=f)
Edit: Ok so I misread your comment and you were talking about >&2. Which is weird because it actually has different behaviour in bash vs zsh. Having said I can't think why you would want the bash behaviour, although the bash behaviour does seem more consistent.It's not particularly surprising that the zsh behavior is different than bash; zsh redirections are only Bourne shell-like in simple cases.
I dislike zsh's behavior because it isn't what the user typed. It should be simple to look at a command line and see how many pipes it makes; see the processes that will be involved in the pipeline. Zsh's behavior seems to me to be clever and implicit; I'd much rather have dumb and explicit.
I'd wondered whether dd or cat were faster, and indeed cat is faster, but not by much. Also, for some embedded devices, you have to write to specific offsets, so dd is more convenient and explicit. Lastly, cat composes poorly with sudo.
$ sudo dd if=foobar.img of=/dev/sdi # works
$ sudo cat foobar.img > /dev/sdi # fails unless root b/c redirection is done by shell
> I only use dd for specific blocks, like writing back a backup of the mbr, or as a rudimentary hex editor.xxd / xxd -r is much nicer, but I suppose sometimes vim is not available...
That performance difference often comes from block size; "dd bs=1M" typically runs much faster than the default block size of 512 bytes.
cat foobar.img | sudo tee /dev/sdi > /dev/null
cat foobar.img | sudo tee /dev/sdi >&-
$ sudo tee /dev/sdi >&- < foobar.img
Or better yet, pv[1]; you even get a progress bar that way![0] https://en.wikipedia.org/wiki/Cat_%28Unix%29#Useless_use_of_...
`od` is ubiquitous--it's POSIX and a requirement of the Single UNIX Specification
sudo (cat foobar.img > /dev/sdi)
Work? sudo sh -c "cat foobar.img > /dev/sdi"
or sudo -s "cat foobar.img > /dev/sdi" sudo sh <<EOF
cat foobar.img > /dev/sdi
EOFIn short, you could do it, but it'd be ripped out of every server that's been hardened, and for users that don't want to care - they're just running 'sudo su' anyhow.
Speaking only for myself, the thought of my shell having a magical escalation process would scare the bejeezus out of me - and I'm supposed to have root on our boxes!
zsh: parse error near `)'
Is it just ignoring the parentheses? Does the redirection "take precedence" over parentheses and brackets?Laptops don't need to follow multiuser best practices, and frankly a complex root password offers little.
sudo (cat foobar.img > /dev/sdi)
No?I am not even sure what you mean by stream or how it would be any difference than "read", "write", "read", "write". Because that is literally what cat does, and so does dd.
Many people feel cat is faster than dd because dd's default block size is 512 bytes as you can see from a simple dd command.
strace dd if=10M of=junk
read(0, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
read(0, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
read(0, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512) = 512
strace cat 10M > junk2
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 131072) = 131072
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 131072) = 131072
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 131072) = 131072
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 131072) = 131072
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 131072) = 131072
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 131072) = 131072
So when benchmarking dd vs cat make sure you are comparing apples to apples.The beauty of dd is you don't have to tinker with things like vm-.dirty_bytes and vm.dirty_background_bytes before using a command. Those should not be messed with on most systems and should definitely not be messed with for the sake of running a single command. When you use cat, it makes some decisions for you. Most of the time makes good choices but it happens to make not so optimal choices for working with large files being written to other devices or file systems.
To avoid the pitfalls of using cat to copy image files (and not have to change global system settings) you use dd. With dd you can use O_DIRECT to bypass the VFS caching layer so your vm.dirty_* settings are ignored in general. You can also specify the block size optimal for the device you are writing to and even reading from.
While dd may not stand for disk or drive or its name have anything to do with disk it is a powerful tool that is the appropriate tool to use when working with large files, precise movement or data and yes copying image files to devices.
<foo.img pv -cNsrc | buffer -p75 -s256k -b128 | pv -cNdst >/dev/whatever
buffer lets the rest of the pipeline proceed while pv is blocked writing to the output file. Yes, conventional filesystem readahead helps to some extent, but IME, not enough, especially when foo.img is a block device itself.
So:
will stream the file rather than what dd does, i.e. read block, write block, read block, write block and so on.Usually I also lower the vm.dirty_bytes and vm.dirty_background_bytes to 16 resp. 48 MB (in bytes) beforehand, which limits the buffer sizes to those amounts. Else it will seem that the progress bar indicates 300MB/s is written, and when it completes you still have to wait a really long time for things to have been written out. Afterwards I restore back vm.dirty_ratio and vm.dirty_background_ratio to respectively 10 and 5 - the defaults on my system.
I wish that all of those projects, tutorials etc. that explain how to write their image to a block device, like an sdcard, would start advise using cat, because there is no reason to use dd, it's just something that people stick with because others do it too.
I only use dd for specific blocks, like writing back a backup of the mbr, or as a rudimentary hex editor.