Counting unique occurences isn't easy if you are given data the way it was described in the article and you have a requirement for exact count.

But there are things you can do to help if you are willing to redesign it a bit.

You could partition the data up front, as it is being written. Then you could send partitions to be processed in parallel (and you still could do this with the shell command!)