Re:R3, a map-reduce engine with Python and Redis
by cypherpunks01 on Thursday July 26, 2012 @10:35PM
(#4299734)
Do the map-reduce results get put back into redis? I always worry about OOM problems when I'm putting a somewhat unbounded set of things into redis.
[ Parent
| on HN ]
Re:R3, a map-reduce engine with Python and Redis
by jdboyd on Friday July 27, 2012 @06:48PM
(#4303935)
You can configure redis to limit how much memory is used.
maxmemory 104857600
Of course, that still might not be the result you want.
Re:R3, a map-reduce engine with Python and Redis
by heynemann on Sunday July 29, 2012 @10:56AM
(#4308615)
They do get put into redis.
Maybe we should have a different storage strategy if the data is too big? File storage? I just meant for it to be simple.
If you are going to use redis for storage then you'll need to fine tune it to the processing you are doing (we have).