News for hackers. Stuff that matters.
Re:Command-line tools for data science by jeroenjanssens on Thursday September 19, 2013 @12:54PM (#6412784)
Suggestions are more than welcome. I'm currently keeping the post up-to-date with suggested tools and repositories.
Re:Command-line tools for data science by susi22 on Thursday September 19, 2013 @02:07PM (#6413221)
http://www.hdfgroup.org/HDF5/doc/RM/Tools.html

http://www.hdfgroup.org/tools5desc.html#1

The first thing I do when I deal with data which has more than --let's say-- 10,000 rows is to put it in HDF file format and work with that. Saves a ton of time while developing a script. I had a python script do a histogram and it ran ~15sec for a file with 100k rows. With converting it first to HDF it ran in ~0.5sec. The import in python is also much shorter (two lines).

HDF is made for high performance numerical I/O. It's great and you can query several structures and even do slices of arrays on the command line (with h5tools).

It's also widely used by Octave, Python, R, Matlab... And you don't have a drawback since you can just pipe it into existing command line tools with a h5dump.

HDF5:

http://www.hdfgroup.org/HDF5/RD100-2002/HDF5_Performance.pdf

http://www.hdfgroup.org/HDF5/RD100-2002/HDF5_Overview.pdf

Re:Command-line tools for data science by twistedpair on Thursday September 19, 2013 @04:22PM (#6414012)
Wow, those benchmarks are old. 195MHz test machine?

Still, 15K rows is not much. Just did that 100K rows (read/sum) bit in Octave. Took about 1/4 second to extract and sum. I assume HDF rocks for much larger sets.

Re:Command-line tools for data science by susi22 on Thursday September 19, 2013 @03:15PM (#6413603)
This also really helpful (pity it's relatively unknown):

https://github.com/dbro/csvquote

And:

http://en.wikipedia.org/wiki/GNU_Core_Utilities (section "Text utilities")

http://directory.fsf.org/wiki/Textutils

Run "$ info coreutils"

Re:Command-line tools for data science by gibrown on Thursday September 19, 2013 @04:25PM (#6414037)
Another shameless plug: http://gibrown.wordpress.com/2013/01/26/unix-bi-grams-tri-gr...

More about Natural Language Processing.

Re:Command-line tools for data science by dirtyvagabond on Thursday September 19, 2013 @02:42PM (#6413407)
nice post! i'd like to make a shameless plug for Drake, a 'make for data' workflow tool. https://github.com/Factual/drake