Re:Realtime Metrics for 128 Million Users with Redis: 50ms +...
by antirez on Saturday March 24, 2012 @06:22AM
(#3749029)
There is an easy fix for this, that is also the recommended way to use bitmaps in Redis: split your bitmap among multiple keys.
For instance you want to set bit i, but you want k bits per every key, the you do:
keyname = "bitmap:"+(i/k)
keybit = i%k
k can be fairly large, like 128k bytes per key. It's still small but big enough for keys overhead to be negligible.
Re:Realtime Metrics for 128 Million Users with Redis: 50ms +...
by firefoxman1 on Saturday March 24, 2012 @01:46PM
(#3750020)
Wow, that's almost exactly how I implemented it:
var bucketSize = 8190;
...
var bucketNumber = Math.floor(userId / bucketSize),
bitInBucket = userId % bucketSize;
...correction on my last comment, looks like I use ~8 thousand bits per bucket, not 8 million.
In my case, and it appears to be true for Spool too, I don't know what bit will be set first. It could be 12 or it could be 2938251, so to prevent a slowdown if the initial bit is a high place I use buckets of bitmaps, each holding around 8 million bits.
[1] Read Warning: http://redis.io/commands/setbit