Re:Useful Unix commands for exploring data
by daurnimator on Wednesday August 27, 2014 @11:00AM
(#8232459)
The utility to do this is called sponge.
http://linux.die.net/man/1/sponge
uniq -u movies.csv | sponge movies.csvsponge is cool. But on debian/ubuntu, it's packaged up in moreutils, which includes a few helpful tools. However a programme called parallel is in moreutils, and that's not as powerful as GNU's parallel. So I often end up uninstalling sponge/moreutils. :(
Re:Useful Unix commands for exploring data
by _delirium on Wednesday August 27, 2014 @11:54AM
(#8232758)
There's some attempt underway to fix that fwiw: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=749355
Re:Useful Unix commands for exploring data
by oftenwrong on Wednesday August 27, 2014 @12:45PM
(#8233079)
GNU Parallel is an indispensable heavy lifter on the command line. I was expecting it to show up in the article.
Re:Useful Unix commands for exploring data
by vacri on Wednesday August 27, 2014 @09:37PM
(#8235543)
moreutils for my usage contains 'chronic', which prepended to a command, stops cron from alerting on any non-error output. Big fan.
Re:Useful Unix commands for exploring data
by yekim on Wednesday August 27, 2014 @11:20AM
(#8232575)
Thanks for throwing this out there. Never heard of this command before! Definitely a good one for my bag o' tricks.
Re:Useful Unix commands for exploring data
by vram22 on Wednesday August 27, 2014 @02:40PM
(#8233919)
The classic book "The UNIX Programming Environment" by Kernighan and Pike, has a tool in it, called 'overwrite', that does this - letting you safely overwrite a file with the result of a oommand or pipeline, IIRC.
Re:Useful Unix commands for exploring data
by danielweber on Wednesday August 27, 2014 @10:31AM
(#8232293)
I came here for that. I learned long ago, the hard way, to never ever use the same file for writing as reading. I was wondering if that rule had changed on me.
Re:Useful Unix commands for exploring data
by aks_c on Wednesday August 27, 2014 @10:36AM
(#8232323)
Thank you for inputs, how about this?
uniq -u movies.csv > temp.csv
temp.csv > movie.csv
rm temp.csv
Re:Useful Unix commands for exploring data
by danielweber on Wednesday August 27, 2014 @01:10PM
(#8233276)
long_running_process > filename.tmp && mv filename.tmp filename
The rename is atomic; anyone opening "filename" will get either the old version, or the new version. (Although it breaks one of my other favorite idioms for monitoring log files, "tail -f filename", because the old inode will never be updated.)
Re:Useful Unix commands for exploring data
by discardorama on Wednesday August 27, 2014 @01:17PM
(#8233321)
> Although it breaks one of my other favorite idioms for monitoring log files, "tail -f filename", because the old inode will never be updated
You should look into the '-F' option of tail; it follows the filename, and not the inode.
you could directly write to uniqMovie.csv in your example. I would do it like below but ONLY once I am certain it is exactly what I want. Usually I just make one clearly named result file per operation without touching the original.
uniq -u movies.csv > /tmp/temp.csv && mv /temp/temp.csv movies.csv
Re:Useful Unix commands for exploring data
by icebraining on Wednesday August 27, 2014 @12:04PM
(#8232842)
$ temp.csv > movie.csv
temp.csv: command not found
You need to be a little careful with that. If you do:
The shell will first open movies.csv for writing (the redirect part) then launch the uniq command connecting stdout to the now emptied movies.csv.Of course when uniq opens movies.csv for consumption, it'll already be empty. There will be no work to do.
There's a couple of options to deal with this, but the temporary intermediate file is my preference provided there's sufficient space - it's easily understood, if someone else comes across the construct in your script, they'll grok it.