Ask HN: Recommend tools to manipulate massive (5 million line) csv files.
Hi HN,
I've been tasked with analyzing a massive dataset and checking for inconsistencies. The data set contains over 5 million lines which makes traditional tools (excel, OOspreadsheet) utterly useless and brings more specialized tools (CSVed) to their knees.
I understand this is well within db territory. In the chance that there are tools out there that can sort and visualize large datasets, I would greatly appreciate your recommendations.
(
10 comments
| 7 points
| on HN
)
Plus, if your comfort level is closer to Excel than MySQL, or you have experience writing Excel macros/userforms, you can do well in Access.
Just promise you will get a proper database if your needs continue to grow; I have seen too many frankensteinian monstrosities grow from once-orderly Access databases!
You may be able to use Tableau Public, but your dataset may exceed the storage limits on Public free account.
Another option may be to use Hadoop-as-a-Service (no visualization) such as Quoble or TreasureData. I recently tried out TreasureData, a decent service if your data has timestamps. You can run HIVE queries to filter and narrow your data and then export out results to use in Visualization softwares.
Shoot me an email ( details in profile ) as I'd love to poke at something just for the line on my resume.
provide more details