News for hackers. Stuff that matters.

A comment on Ask HN: What are some ways to work with large amounts of data quickly?.

Re:Ask HN: What are some ways to work with large amounts of ... by realtarget on Monday June 06, 2016 @03:08AM (#11845130)
What do you mean by "large amounts of data"?

A friend of mine is a data analyst at a large market research company and analyzes more than 10 billion cookies a month using Hadoop from apache: http://hadoop.apache.org/

[ Parent | on HN ]
Re:Ask HN: What are some ways to work with large amounts of ... by nstart on Monday June 06, 2016 @03:53AM (#11845257)
well let's say you want to create a minute by minute report on URL's visited in an ecommerce site that has a large amount of usage. Every minute I want to process all those URLs and decide if I want to create a flash sale on the site. Since this was an example I just thought of, let's imagine that there are 400k URLs visited every minute.

That would be a decent definition of what I meant by large amounts of data. Sorry for not making that clearer before.

Your friend's example is pretty good. Curious how they work on the data in memory.

[ Parent | on HN ]
Re:Ask HN: What are some ways to work with large amounts of ... by ahoka on Monday June 06, 2016 @06:53AM (#11845794)
It depends on how many unique URLs you have. If you have like a few hundreds or even thousand I would just use a simple hash table with counters to see what's a hot sale right now (I guess that's what you want in your example). If you have several millions than maybe I would put it into Redis or something similar where I can utilize ready-to-use HyperLogLog cardinality estimation.
[ Parent | on HN ]