TSV Utilities: Command line tools for large, tabular data files

Re:TSV Utilities: Command line tools for large, tabular data... by dima55 on Saturday August 31, 2019 @05:43PM (#20848953)

There're lots of tools in this space that are similar in a very general way, but have widely different design choices. My tookit is

https://github.com/dkogan/vnlog

It's tsv-utils-like, but is strictly a wrapper around existing tools. So filtering and transformations are interpreted literally as awk (or perl) expressions. And the various cmdline options match the standard tool options because they ARE the standard tools. So you get a very friendly learning curve, but something like tsv-utils is probably faster and probably more powerful. And it looks like tsv-utils references fields by number instead of by name. Many of the others (mine included) use the field names, which makes a MAJOR usability improvement.

Other tools in no particular order:

https://csvkit.readthedocs.io/

https://github.com/johnkerl/miller

https://github.com/eBay/tsv-utils-dlang

http://harelba.github.io/q/

https://github.com/BatchLabs/charlatan

https://github.com/dinedal/textql

https://github.com/BurntSushi/xsv

https://github.com/dbohdan/sqawk

https://stedolan.github.io/jq/

https://github.com/benbernard/RecordStream

[ Parent | on HN ]

Re:TSV Utilities: Command line tools for large, tabular data... by cube2222 on Sunday September 01, 2019 @05:47AM (#20851401)

Adding another one which I'm one of the authors of: https://github.com/cube2222/octosql/

It tries to provide ergonomic data munching of various formats, but using a sql interface, which most will probably feel immediately at home with.

[ Parent | on HN ]

Re:TSV Utilities: Command line tools for large, tabular data... by bradknowles on Monday September 02, 2019 @11:53AM (#20859832)

With respect, who actually feels at home with SQL?

It's about the most alien and obtuse language I've ever had the misfortune of encountering, and in that category I rate it worse than COBOL, FORTRAN, Assembly, C, Prolog, Sendmail re-write rules, BASH, and every other language I've ever encountered and had to use, but which I cannot recall at the moment.

[ Parent | on HN ]

Re:TSV Utilities: Command line tools for large, tabular data... by he_the_great on Wednesday September 04, 2019 @10:36AM (#20877560)

SQL gets rather mistifying when you move towards some complex joins and data mapping. PSQL can also feel Sadly out of place.

But their is real nice clarit in what you want at its foundation.

[ Parent | on HN ]

Re:TSV Utilities: Command line tools for large, tabular data... by veritas3241 on Tuesday September 03, 2019 @08:32PM (#20872819)

I am! SQL is the language of data and analytics and it can be quite a joy to use. But maybe I've just Stockholm'd myself into liking it.

[ Parent | on HN ]

Re:TSV Utilities: Command line tools for large, tabular data... by dzamo_norton on Monday September 09, 2019 @09:17AM (#20917816)

> With respect, who actually feels at home with SQL?

So many data analysts that each time a new analytical database engine comes out it needs to support SQL or something like it to get any mindshare.

[ Parent | on HN ]

Re:TSV Utilities: Command line tools for large, tabular data... by j88439h84 on Sunday September 01, 2019 @01:41AM (#20850660)

Another one, in Python:

https://github.com/python-mario/mario

[ Parent | on HN ]