I enjoyed this. Would most large scale applications of these ideas happen with Hadoop? Any other software I should be aware of for, e.g., estimating frequencies in a large text stream?