News for hackers. Stuff that matters.
Re:Pipe Viewer by spudlyo on Thursday May 24, 2012 @04:35PM (#4020402)
It's also great that pv supports rate limiting, so you can choose if you want to saturate your network when you use it with netcat.
[ Parent | on HN ]
Re:Pipe Viewer by there on Thursday May 24, 2012 @11:38PM (#4021778)
Somewhat related, trickle is a good user-space bandwidth shaper if you aren't sending data over a pipe: http://monkey.org/~marius/pages/?page=trickle
[ Parent | on HN ]
Re:Pipe Viewer by dbuthay on Thursday May 24, 2012 @04:16PM (#4020339)
It's amazingly useful to have realtime throughput rates

tail -f query.log | pv -tri 1 > /dev/null

[ Parent | on HN ]
Re:Pipe Viewer by ralph on Thursday May 24, 2012 @05:04PM (#4020555)
Note, a plain dd will report progress on stderr if sent a SIGUSR1. That can sometimes be enough to avoid the overhead of pv(1) and sending huge amounts of data through another pipe.
[ Parent | on HN ]
Re:Pipe Viewer by protopete on Thursday May 24, 2012 @05:32PM (#4020683)
About the pv overhead, I sent a note to the pv author saying that on Linux at least the transfer function could take advantage of the splice system call. Splice "copies" data from one file descriptor to another file descriptor, avoiding having to copy the data from the kernel to userspace and back.
[ Parent | on HN ]
Re:Pipe Viewer by Erwin on Thursday May 24, 2012 @07:36PM (#4021149)
On FreeBSD (and maybe Mac OS X) you can press Control-T to send a SIGINFO signal (non-standard extension. This prints out how far dd has gotten (and perhaps other FreeBSD utilities too).
[ Parent | on HN ]
Re:Pipe Viewer by there on Thursday May 24, 2012 @11:35PM (#4021765)
You may need to setup your terminal to do that with:

    stty status '^T'
fsck also responds to SIGINFO by printing its status. I had to hack up rsync to do it (http://jcs.org/patches/rsync-show_current_file.diff), and I wish more utilities supported it by default.
[ Parent | on HN ]
Re:Pipe Viewer by sneak on Friday May 25, 2012 @05:03AM (#4022659)
N.B. also that dd on osx will EXIT IMMEDIATELY if sent a SIGUSR1. It double-sucks because one tends to do this on long-running dd processes that you're wondering are done yet...
[ Parent | on HN ]
Re:Pipe Viewer by mibbitier on Thursday May 24, 2012 @04:04PM (#4020277)
How does it know the progress / ETA?

I was under the assumption that pipes do not know how much total data is on its way?

[ Parent | on HN ]
Re:Pipe Viewer by aidenn0 on Thursday May 24, 2012 @04:05PM (#4020287)
If you give it a file, or specify the amount on the commandline it knows, otherwise it doesn't
[ Parent | on HN ]
Re:Pipe Viewer by mbreese on Thursday May 24, 2012 @04:14PM (#4020334)
Right. I usually use pv in place of 'cat' at the start of a long set of piped commands. Anywhere you could use 'cat' you can use pv. If I'm not mistaken, I think you can also use multiple 'pv' commands at a time too.
[ Parent | on HN ]
Re:Pipe Viewer by pronoiac on Thursday May 24, 2012 @05:28PM (#4020663)
> If I'm not mistaken, I think you can also use multiple 'pv' commands at a time too.

I use multiple pv instances to track compression/decompression rates, and overall progress, for tarballs. Using the "-c" code for console codes keeps the instances from stomping over each other's output.

[ Parent | on HN ]
Re:Pipe Viewer by lloeki on Thursday May 24, 2012 @05:45PM (#4020740)
This is indeed visible in the example, only source has progress while bzcat and gzip only have flow.
[ Parent | on HN ]
Re:Pipe Viewer by bsg75 on Thursday May 24, 2012 @04:08PM (#4020304)
You have to supply it the stream sizes if you want completion percentages, otherwise it displays rates only.

Example: https://gist.github.com/2783910

[ Parent | on HN ]
Re:Pipe Viewer by b0b0b0b on Thursday May 24, 2012 @10:14PM (#4021569)
If I'm dealing with [compressed] files on disk or across a network, what's the practical overhead of pv? A few extra percent CPU, or some material decrease in through put?
[ Parent | on HN ]
Re:Pipe Viewer by blibble on Friday May 25, 2012 @09:38AM (#4023380)
very very little, it's just doing read(), counting the bytes, then write(), with some screen output every few seconds.

(hell it could even use splice() and never even see the bytes)

[ Parent | on HN ]
Re:Pipe Viewer by crazy_eye on Thursday May 24, 2012 @03:44PM (#4020185)
love pv, mostly use it for percentage bar on db restores:

pv -f *.sql.gz | zcat | mysql db_instance_name

[ Parent | on HN ]
Re:Pipe Viewer by pramodbiligiri on Thursday May 24, 2012 @04:18PM (#4020346)
Nifty command. Been using this for a month or so. One use case is measuring if a downstream process can consume data as fast the previous process is producing it (by alternating second process with /dev/null or some such).
[ Parent | on HN ]
Re:Pipe Viewer by bsg75 on Thursday May 24, 2012 @03:44PM (#4020182)
Filed under "How did I not know this?!", with credit to @hmason on Twitter.
[ Parent | on HN ]
Re:Pipe Viewer by pyre on Friday May 25, 2012 @08:19AM (#4023156)
It's been here before:

http://news.ycombinator.com/item?id=3626020

http://news.ycombinator.com/item?id=3198572

I'm sure there were older stories too. IIRC I saw it on HN longer than 6 months ago, but I couldn't find that submission in a cursory search.

[ Parent | on HN ]
Re:Pipe Viewer by gouranga on Thursday May 24, 2012 @04:45PM (#4020453)
Nice - that'll spice up my rather boring svnadmin dump sessions - thanks for posting!
[ Parent | on HN ]
Re:Pipe Viewer by kilovoltaire on Thursday May 24, 2012 @07:43PM (#4021175)
yes | pv > /dev/null
[ Parent | on HN ]
Re:Pipe Viewer by sp332 on Thursday May 24, 2012 @08:17PM (#4021271) 

  pv /dev/zero > /dev/null
[ Parent | on HN ]
Re:Pipe Viewer by kilovoltaire on Friday May 25, 2012 @09:48AM (#4023421)
awesome, i was actually wondering if there was a "yes-esque" (yesque?) /dev/ file.

my throughput from yes is 49MB/s whereas from /dev/zero it's 12GB/s

[ Parent | on HN ]
Re:Pipe Viewer by potatohead00 on Thursday May 24, 2012 @09:20PM (#4021431)
Found out about pv ~year ago when working with large log files. Very useful. I also like it for testing how quickly a script can consume data.
[ Parent | on HN ]