News for hackers. Stuff that matters.
Re:Ask HN: What are the best tools for analyzing large bodie... by codeonfire on Wednesday June 17, 2015 @10:15PM (#9736439)
If you want high performance and simple why not use flat files, bash, grep (maybe parallel), cut, awk, wc, uniq, etc. You can get very far with these and if you have a fast local disk you cat get huge read rates. A few GB can be scanned in a matter of seconds. Awk can be used to write your queries. I don't understand what you are trying to do, but if it can be done with a SQL database and doesn't involve lots of joins then it can be done with a delimited text file. If you don't have a lot of disk space you can also work with gzipped files, zcat, zgrep, etc. I would not even consider distributed solutions or nosql until I had at least 100GB of data (more like 1TB of data). I would not consider any sort of SQL database unless I had a lot of complex joins.
Re:Ask HN: What are the best tools for analyzing large bodie... by cactusface on Thursday June 18, 2015 @01:41AM (#9737001)
This is definitely the most straightforward way to go. It is almost certainly not the way he will go. In particular Awk is just great for this job and very easy to learn.