Re:Awk: `Begin { ` Part 1
by charris000 on Friday October 30, 2020 @11:07AM
(#24942984)
+1 for Miller. It's a really slick tool for exploring large csv datasets. I've typically used it to do some prototyping and exploration of 200-500GB csv datasets before doing more hefty work in Java+PigLatin (our use-case is more long-term than just a single run for analysis, so that's re reason for moving out of just Miller). It's great to get a feel for the data and run some initial analysis before diving into the larger, more cumbersome system and tooling.
Re:Awk: `Begin { ` Part 1
by Gravityloss on Friday October 30, 2020 @08:06AM
(#24941346)
This seems very interesting.
Making this reproducable and usable for multiple people has always been an annoyance in semi-ad-hoc data collection.
For example analysing the number of people per-year over multiple differently formatted files is as easy as `mlr uniq -f pid,year then count -g year`. It has filtering, very extensive manipulation capabilities and a nice documentation.
John has also reacted very fast on feature requests I had (so fast that I've not yet implemented using them).
[1]: https://miller.readthedocs.io/en/latest/#overview