TSV Utilities: Command line tools for large, tabular data files

Re:TSV Utilities: Command line tools for large, tabular data... by IanCal on Saturday August 31, 2019 @06:34PM (#20849224)

Big fan of visidata https://visidata.org/ - one of those tools that does a lot but easily provides useful things with little complexity. For example open, select column, shift-f gives a histogram. Enter on one of those rows filters the data to just them.

Can load a variety of data formats + anything pandas can (so parquet works).

Edit - you can do a lot with it, quickly creating columns with python expressions, plots, joining tables, etc. but it's great for things like "pipe in kubectl output - get numbers of each job status, wait why are 10 failed, filter to those..." which easily works with any fixed width output.

[ Parent | on HN ]

Re:TSV Utilities: Command line tools for large, tabular data... by austinjp on Sunday September 01, 2019 @04:03AM (#20851076)

Thanks for this. Looks perfect for my needs. I've got to analyse a hideous generated CSV file with thousands of columns. I was planning to load it into SQLite3 but it has too many columns for import. I was also unpleasantly surprised to find that the column names were too long for PostegresQL too. Python + pandas can handle it, but a tool for quick interactive exploration is just what I need. Hopefully visidata will provide!

[ Parent | on HN ]

Re:TSV Utilities: Command line tools for large, tabular data... by IanCal on Sunday September 01, 2019 @05:43AM (#20851389)

If you still hit size issues, you might be able to throw things through xsv first (list columns, select just a few columns and pipe out, get stats on contents of columns, samples, etc). It's one of my other goto things https://github.com/BurntSushi/xsv

[ Parent | on HN ]

Re:TSV Utilities: Command line tools for large, tabular data... by DSingularity on Saturday August 31, 2019 @07:04PM (#20849366)

I’ve been looking for exactly this!

[ Parent | on HN ]