News for hackers. Stuff that matters.
Re:Shell Magic: Set Operations with uniq by jph on Tuesday May 29, 2018 @08:52PM (#17184055)
I like shell set operations scripts, because they are quick and easy.

I prefer `awk` over `uniq` and `comm` because awk tends to be faster at set ops that can skip sorting and deduplicating.

Here's my script for union, intersection, etc. See README on GitHub. Suggestions welcome.

https://github.com/sixarm/setop

    #!/bin/sh
    set -eu
    op="$1"; shift

    case  $op  in
      ∪|u|union|or|∨|add|addition|'+'|'|')
        awk '!seen[$0] {print} {seen[$0]=1}' "$@"
        ;;
      ∩|i|intersection|and|∧|'&')
        awk 'FNR==1{argind+=1} seen[$0]+=1 {next} END { for (key in seen) { if (seen[key]==argind) { print key } } }' "$@"
        ;;
      ⊖|d|diff|difference|xor|⊻)
        awk 'FNR==1{argind+=1} seen[$0]+=1 {next} END { for (key in seen) { if (seen[key]==1) { print key } } }' "$@"
        ;;
      ex|except|exclude|subtract|subtraction|'-')
        awk 'NR==FNR{seen[$0]=1;next} seen[$0]=0; END { for (key in seen) { if (seen[key]) { print key } } }' "$@"
        ;;
      extra)
        awk 'BEGIN{argindmax=ARGC-1} FNR==1{argind+=1} argind<argindmax {seen[$0]; next}!($0 in seen)' "$@"
        ;;
      disjoint|n|not|none)
        awk 'seen[$0]==1 {x=1;exit} {seen[$0]=1} END { print x ? "FALSE" : "TRUE"}; exit !!x}' "$@"
        ;;
      *)
    esac
Re:Shell Magic: Set Operations with uniq by unixhero on Wednesday May 30, 2018 @05:11AM (#17186222)
A few comments. First, wow, I love how this looks like dark magic.

Those mathematical notations, are you using them because it makes it easier to see how it corresponds to actual Set Theory/theorems? If so, could you just as well have used an alphanummeric identifier like "left" "union" "right" or - would the code break without this notation? I'm on deep waters here, I don't know this. But set theory seems to pop up a lot in my line of work, essentially doing joins in datasets using Tableau - so my interest in the nitty gritty of this field is increasing.

Re:Shell Magic: Set Operations with uniq by jph on Wednesday May 30, 2018 @01:45PM (#17189829)
Thanks! To answer your questions...

> because it makes it easier to see how it corresponds to actual Set Theory/theorems?

Yes. These are the Unicode symbols for set theory.

> could you just as well have used an alphanummeric identifier like "left" "union" "right" or

Yes. You can use any of the words in the case switch statements, such as `setop union file1 file2`. You can also edit the script to add your own words if you like.

You can see simpler versions of these scripts in our GitHub repos. For example the `union` command is https://github.com/sixarm/union

> set theory seems to pop up a lot in my line of work

More and more in mine too. Thank you for your comments!

Re:Shell Magic: Set Operations with uniq by FractalLP on Tuesday May 29, 2018 @09:53PM (#17184401)
How long have you been using Awk?
Re:Shell Magic: Set Operations with uniq by jph on Tuesday May 29, 2018 @10:20PM (#17184533)
For a few years for two clients (financial and governmental) that have a range of older controlled systems.

We benefit from simple analytics that are POSIX, not python, perl, R, etc. http://www.numcommand.com/

Re:Shell Magic: Set Operations with uniq by FractalLP on Wednesday May 30, 2018 @06:14PM (#17192372)
I've never heard of numcommand before, but it looks super awesome and exactly what I need. I use Linux servers, but am forced to use Windows as my local. So first question, do you have experience with awk/gawk on Windows? Second question: What do you prefer about Awk over Python/Perl? I guess if all your analytics are fairly short it is just easier in Awk? Thanks!
Re:Shell Magic: Set Operations with uniq by jph on Thursday May 31, 2018 @10:31AM (#17196565)
Thank you!

Yes in my experience of awk/gawk on Windows, things work the same, other than file separators, line endings, and similar platform-specific issues.

For your second question, awk simply works. It's reliable, fast, and everywhere. I also like python and perl, and choose these for any modern system.

Re:Shell Magic: Set Operations with uniq by FractalLP on Friday June 01, 2018 @10:41PM (#17212330)
Thank you!
Re:Shell Magic: Set Operations with uniq by unixhero on Wednesday May 30, 2018 @05:13AM (#17186229)
'Hat tip
Re:Shell Magic: Set Operations with uniq by int0x80 on Wednesday May 30, 2018 @02:06AM (#17185448)
Thats nice, so is the article's examples. But what I always need to lookup is how to make the operations work on especific columns, rather than on whole lines.

I mean without resorting to awk.

Re:Shell Magic: Set Operations with uniq by fiddlerwoaroof on Wednesday May 30, 2018 @12:29AM (#17185075)
doesn't this read the entire input into memory? `uniq` and `comm` don't (need to) do this, so they can work on inputs bigger than available memory.
Re:Shell Magic: Set Operations with uniq by jph on Wednesday May 30, 2018 @01:54PM (#17189919)
Yes in the `setop` current implementation, because this enables the script to be fast, short, and work with inputs without needing to call `sort`.

For comparison, a typical POSIX `uniq` implementation reads the input and solely compares two adjacent lines; this requires the input to be presorted.

An interesting upgrade could be to add a `setop` option flag that tells the script the inputs are already sorted and/or deduped. This can achieve the memory savings you're describing.

Re:Shell Magic: Set Operations with uniq by amelius on Wednesday May 30, 2018 @07:05AM (#17186660)
Could you add a "--help" option? Or provide usage instructions in a comment?