News for hackers. Stuff that matters.
Re:Realtime Metrics for 128 Million Users with Redis: 50ms +... by firefoxman1 on Friday March 23, 2012 @11:47PM (#3748419)
This was posted a while ago, and I have since implemented bitmaps myself. One thing I learned from the documentation[1] is that setting an initial bit at a very high-numbered place (like 2^30 - 1) takes a while to allocate (compared to Redis's normal speed) and blocks other operations in the process.

In my case, and it appears to be true for Spool too, I don't know what bit will be set first. It could be 12 or it could be 2938251, so to prevent a slowdown if the initial bit is a high place I use buckets of bitmaps, each holding around 8 million bits.

[1] Read Warning: http://redis.io/commands/setbit

Re:Realtime Metrics for 128 Million Users with Redis: 50ms +... by antirez on Saturday March 24, 2012 @06:22AM (#3749029)
There is an easy fix for this, that is also the recommended way to use bitmaps in Redis: split your bitmap among multiple keys.

For instance you want to set bit i, but you want k bits per every key, the you do:

    keyname = "bitmap:"+(i/k)
    keybit = i%k
k can be fairly large, like 128k bytes per key. It's still small but big enough for keys overhead to be negligible.
Re:Realtime Metrics for 128 Million Users with Redis: 50ms +... by firefoxman1 on Saturday March 24, 2012 @01:46PM (#3750020)
Wow, that's almost exactly how I implemented it:

   var bucketSize = 8190;

   ...

   var bucketNumber = Math.floor(userId / bucketSize),
       bitInBucket = userId % bucketSize;

...correction on my last comment, looks like I use ~8 thousand bits per bucket, not 8 million.