Re:Useful Unix commands for exploring data
by pselbert on Wednesday August 27, 2014 @11:58AM
(#8232790)
You don't have to read all of a file into memory in Ruby. There are a number of facilities for reading only a portion of a file, readpartial[1] for example. Additionally, you have access to all of the native pipe[2] functionality as well. There are plenty of reasons to favor shell tools over Ruby, but those aren't some of them.
[1]: http://www.ruby-doc.org/core-2.1.2/IO.html#method-i-readpart... [2]: http://www.ruby-doc.org/core-2.1.2/IO.html#method-c-popen
Re:Useful Unix commands for exploring data
by dpeck on Wednesday August 27, 2014 @12:21PM
(#8232947)
No the case with ruby at all, if you're reading the whole file into memory theres a good chance you're doing it wrong.
check out yield and blocks
Re:Useful Unix commands for exploring data
by barrkel on Wednesday August 27, 2014 @01:03PM
(#8233227)
The problem is that the most obvious way of doing it - File.readlines('foo.txt').map { ... }.select { ... } etc. - is not stream-oriented.
Re:Useful Unix commands for exploring data
by riffraff on Wednesday August 27, 2014 @04:59PM
(#8234684)
arguably, it's trivial to make that stream oriented
open('tmp.rb').each_line.lazy.map {...}.select {...}
Re:Useful Unix commands for exploring data
by barrkel on Thursday August 28, 2014 @05:19AM
(#8236493)
> open('tmp.rb').each_line.lazy
NoMethodError: undefined method `lazy' for #<Enumerator: #<File:Procfile>:each_line>
For example, working with file lines naively in Ruby means reading the whole lot into a giant array and doing transformations an array at a time, rather than in a streaming fashion.
The shell gives you fairly safe concurrency and streaming for free.
Personally, if it's a complex task, I generally write a tool such that it can be put into a shell pipeline.
Knowing the command line well - so that you don't often have to look up man pages for obscure flags / functionality - has its own rewards, as these commands turn into something you use all the time in the terminal. Rather than spending a few minutes developing a script in an editor, you can incrementally build a pipeline over a few seconds. Doing your script in a REPL is a better approximation, but it's a bit less immediate.