News for hackers. Stuff that matters.
Re:View Counting at Reddit by bognition on Saturday May 27, 2017 @05:14PM (#14432007)
How do you concurrently update a counter?
Re:View Counting at Reddit by eriknstr on Saturday May 27, 2017 @05:29PM (#14432057)
I think the main problem is not concurrently updating the counter. After all updating the HLL must also be done concurrently.

The primary motivation on their part for using HLL is given in the intro.

>In order to maintain an exact count in real time we would need to know whether or not a specific user visited the post before. To know that information, we would need to store the set of users who had previously visited each post, and then check that set every time we processed a new view on a post. A naive implementation of this solution would be to store the unique user set as a hash table in memory, with the post ID as the key.

>This approach works well for less trafficked posts, but is very difficult to scale once a post becomes popular and the number of viewers rapidly increases. Several popular posts have over one million unique viewers! On posts like these, it becomes extremely taxing on both memory and CPU to store all the IDs and do frequent lookups into the set to see if someone has already visited before.

Re:View Counting at Reddit by BillinghamJ on Saturday May 27, 2017 @05:26PM (#14432045)
Redis writes are atomic - you just use the increment function
Re:View Counting at Reddit by bognition on Saturday May 27, 2017 @07:29PM (#14432420)
Writes are atomic in redis because redis is single threaded. So you are bounded by how fast redis can write. If you try to write any faster then redis can handle you'll get queueing or errors.
Re:View Counting at Reddit by antirez on Saturday May 27, 2017 @11:18PM (#14432955)
The wonderful thing about HyperLogLogs is that you can split the counter in N servers and "merge" the registers later, in case you want an architecture that shards the same counter in multiple servers. But sharding directly by resource looks simpler actually...
Re:View Counting at Reddit by btmorex on Saturday May 27, 2017 @07:50PM (#14432480)
Run enough redis servers to handle the load. Choose a server by hashing a user id. Total = sum of counts from all servers.
Re:View Counting at Reddit by jjirsa on Tuesday May 30, 2017 @12:21AM (#14442990)
It's also got cassandra behind the scenes, which has fast, concurrent, distributed counters.