Re:From 1.5 GB to 50 MB: The Story of my Redis Database
by ajtaylor on Thursday March 21, 2013 @11:23AM
(#5415646)
I was thinking you had switched from string keys to hashes, since I had read a post from antirez about hashes being more efficient for this type of thing. But as they say, measure twice, cut once. After all, profiling is the first step of optimisation.
[ Parent
| on HN ]
Re:From 1.5 GB to 50 MB: The Story of my Redis Database
by ihsw on Thursday March 21, 2013 @02:46PM
(#5417740)
Hashes can be efficient because redis compresses hashes that match certain restrictions (eg: under 1000 items, each item being under 1000 characters long) and the same can apply to lists, sets, and zsets.
This compression may have inadvertently been activated by the author when he pruned his zsets, which is why the savings are so apparent.
Re:From 1.5 GB to 50 MB: The Story of my Redis Database
by ajtaylor on Thursday March 21, 2013 @10:44PM
(#5421063)
The primary reason for the reduction was going from ~7k items in the ZSET to 100 for several thousand ZSETS. That's not to say the built-in compression didn't affect anything, but obviously less data == less memory.
Re:From 1.5 GB to 50 MB: The Story of my Redis Database
by davidcelis on Thursday March 21, 2013 @11:28AM
(#5415700)
This was one of the frustrating parts of trying to fix my issue. Every search I did for Redis optimization tips said to use hashes to reduce redundant keys (and, in doing so, use a memory efficient data structure). But every key my library uses is a SET or ZSET so, of course, hashes were off the table for me.
Re:From 1.5 GB to 50 MB: The Story of my Redis Database
by ihsw on Thursday March 21, 2013 @02:48PM
(#5417761)
The optimization trick is outlined here:
http://redis.io/topics/memory-optimization
If your sets and zsets are bounded within a certain limit then they will be compressed and take up quite a bit less space.
Updating your configuration obviously requires a restart so that will need to be kept in mind (ie: if your data fragmentation is high then restarting will eliminate the fragmentation, so space savings will be apparent but they're not actually indicative of any changes).