I imagine HN comments would have a ton of tags generated by such an approach. So, a straight-forward search engine for them could help too. Not a Big Data guy so I can't do recommendations. However, this DARPA-funded, Watson alternative was open-sourced and might do interesting things for HN data:
1. Filter out anything below 5 upvotes instantly.
2. Merge comments of duplicate OP's into one super-post.
3. Provide Vivisimo-like tech to auto-categorize and search those. Nice picture of what that looked like in document below.
https://faculty.ist.psu.edu/jjansen/academic/pubs/vivisimo_j...
I imagine HN comments would have a ton of tags generated by such an approach. So, a straight-forward search engine for them could help too. Not a Big Data guy so I can't do recommendations. However, this DARPA-funded, Watson alternative was open-sourced and might do interesting things for HN data:
http://deepdive.stanford.edu/