News for hackers. Stuff that matters.
Re:How to Clean Text Data at the Command Line by jhoechtl on Sunday October 18, 2020 @11:14AM (#24818430)
I beg to differ. I did a lot of csv wrangling on Unix. Csv is a beast. My tools of choice ultimately was miller, an absolutely underrated tool:

https://github.com/johnkerl/miller

Re:How to Clean Text Data at the Command Line by leonim on Sunday October 18, 2020 @01:37PM (#24819600)
Totally agree. I think mlr is a wonderful CLI tool. It is very robust, and can handle/convert multiple tabular formats including csv, tsv, json, fixed-format, etc. It has a pretty decent text output formatting, with the --opprint flag.

I use to be very comfortable using awk/sed/perl/sort/uniq/tr/tail/head from the CLI for the sort of data cleaning this article is talking about. However, over the past year I've found I use VisiData https://github.com/saulpw/visidata for interactive work.

If I need to clean up the data first, I'll use mlr or jq as input to Visidata. If my data is too dirty for mlr, then I'll use Unix toolbox tools mentioned as input to mlr, jq or VisiData.

VisiData provides some ability to script, but when possible I prefer to have the shell do the scripting with all the tools mentioned as input to Visidata.

Re:How to Clean Text Data at the Command Line by nicoburns on Sunday October 18, 2020 @11:55AM (#24818757)
Oo, I've not come across this one before. Looks useful! In a similar vein of CSV tools that seem to be underrated, I recently came across Daff which does excellent tabular diffs.

https://github.com/paulfitz/daff