Re:Ask HN: What are some ways to work with large amounts of ...
by ahoka on Monday June 06, 2016 @06:46AM
(#11845759)
It usually boils down to the following simple principles and techniques:
A: Probabilistic data structures and algorithms
For example you can quickly estimate if a very large set contains a certain key with bloom filters or estimate cardinality with hyperloglog.
B: Divide and conquer
Just divide your data to workable pieces and combine partial results at the end. Hadoop is a popular infrastructure for doing this. Other example is doing pieces of work in memory and saving it to disk by chunks and reducing it later to get the final result.
A: Probabilistic data structures and algorithms For example you can quickly estimate if a very large set contains a certain key with bloom filters or estimate cardinality with hyperloglog.
B: Divide and conquer Just divide your data to workable pieces and combine partial results at the end. Hadoop is a popular infrastructure for doing this. Other example is doing pieces of work in memory and saving it to disk by chunks and reducing it later to get the final result.