Re:Realtime Metrics for 128 Million Users with Redis: 50ms +...
by firefoxman1 on Friday March 23, 2012 @11:47PM
(#3748419)
This was posted a while ago, and I have since implemented bitmaps myself. One thing I learned from the documentation[1] is that setting an initial bit at a very high-numbered place (like 2^30 - 1) takes a while to allocate (compared to Redis's normal speed) and blocks other operations in the process.
In my case, and it appears to be true for Spool too, I don't know what bit will be set first. It could be 12 or it could be 2938251, so to prevent a slowdown if the initial bit is a high place I use buckets of bitmaps, each holding around 8 million bits.
Re:Realtime Metrics for 128 Million Users with Redis: 50ms +...
by mattlong on Friday March 23, 2012 @11:30PM
(#3748384)
I know disk space is cheap these days, but at 16MB/metric/<level-of-granularity>, it seems like your metric dataset would grow pretty quick. With just 10 metrics tracked daily, thats another gigabyte per week. Of course it does come with the benefit of maintaining all the raw data since you never roll up or aggregate the data...so the pros probably outweigh that con. :)
Re:Realtime Metrics for 128 Million Users with Redis: 50ms +...
by kijin on Saturday March 24, 2012 @12:10AM
(#3748454)
Redis stores everything in RAM, and RAM is not as cheap as disk. Adding GB's of RAM every week will quickly get rather expensive. But I guess you could dump old data to disk and load it back to Redis only when you need it. It might even compress well, depending on what the metrics track.
Re:Realtime Metrics for 128 Million Users with Redis: 50ms +...
by Erwin on Saturday March 24, 2012 @04:05AM
(#3748834)
Isn't Redis still single-threaded for queries, but saving in the background? That seems a little risky: you've got your 100 million users setting bits in your bitsets and suddenly everything blocks for 10 seconds while old data is being loaded from disk.
Re:Realtime Metrics for 128 Million Users with Redis: 50ms +...
by rythie on Saturday March 24, 2012 @03:00PM
(#3750281)
It only really needs be captured like that using redis, when the collection period is up you could store it on disk, or even just the aggregate information on disk.
Re:Realtime Metrics for 128 Million Users with Redis: 50ms +...
by mattparlane on Saturday March 24, 2012 @04:46AM
(#3748882)
The only problem with this method is that it requires that IDs are integers, start at 1 and increment by 1.
I'm using MongoDB and IDs are 12-byte values of which the first four are a timestamp. Does anyone know of a way to make this method work, ideally without adding another field to the collection?
Re:Realtime Metrics for 128 Million Users with Redis: 50ms +...
by simonw on Saturday March 24, 2012 @06:53AM
(#3749052)
The comments on the article address this - the OP is using UUIDs as the primary key for their users, but each user is also assigned an "analytics key" which is an integer that started at one. You can even use the redis INCR command to generate these on demand.
Re:Realtime Metrics for 128 Million Users with Redis: 50ms +...
by bluemoon on Saturday March 24, 2012 @07:23AM
(#3749090)
There is a way in mongo to replace the id with an auto incrementing number. Have a look at the docs. It's also helpful of you want to use the id as a base62 value for urls
Re:Realtime Metrics for 128 Million Users with Redis: 50ms +...
by ericd on Saturday March 24, 2012 @06:06AM
(#3749001)
I still think that those user numbers in the title evokes a mental image of a certain type of load with a certain type of response time. I think if you got rid of the response time, it would be less linkbaity, because then it's clear that your focus is on the amount of storage it would take. It's not very important either way.
In my case, and it appears to be true for Spool too, I don't know what bit will be set first. It could be 12 or it could be 2938251, so to prevent a slowdown if the initial bit is a high place I use buckets of bitmaps, each holding around 8 million bits.
[1] Read Warning: http://redis.io/commands/setbit
For instance you want to set bit i, but you want k bits per every key, the you do:
k can be fairly large, like 128k bytes per key. It's still small but big enough for keys overhead to be negligible.2 million users' actions could be tracked in 250k per metric. 10 metrics per day is 2.5m per day x 7 days is back to just over 16m (17.5m).
Redis has a mode which does this automatically I believe (and it's the default if I remember correctly).
I'm using MongoDB and IDs are 12-byte values of which the first four are a timestamp. Does anyone know of a way to make this method work, ideally without adding another field to the collection?
I hope it's clearer now.
was:
is: