News for hackers. Stuff that matters.
Re:Having fun with Redis Replication between Amazon and Rack... by antirez on Tuesday July 31, 2012 @11:31AM (#4317732)
With a 2.6 system this problems will be simpler to detect, because at this point we have explicit (configurable) limits for clients, of type normal, pubsub, slaves, so the connection is dropped if the amount of pending data goes over a given limit (and the event logged).

This added a bit of complexity to a few code paths that now require to estimate the size of objects, data, and so forth, but I think this will pay back in the long run.

This blog post makes me wonder if we should implement compression in the replication link as a feature in the future, or at least binary encoding. In short there is a plan for a binary version of AOF / replication-link that can save a lot of bandwidth.

Re:Having fun with Redis Replication between Amazon and Rack... by solso on Tuesday July 31, 2012 @11:39AM (#4317785)
woops! @antirez in person :-) amazing job you have done with Redis.

Initially we didn't add compression to save bandwidth but to solve the network bandwidth issues between amazon and rackspace. However, it turns out that you can save on the bill quite a lot, in our case about ~ $700.

Regarding the binary, would be very nice to have. The first though is that compression would work better for us than binary since our keys are quite long (50/100 bytes) and very regular on their patterns, so compression works very well in our case. Probably can be extrapolated for other cases where the keys are typically heavier than the values.

Re:Having fun with Redis Replication between Amazon and Rack... by Smrchy on Tuesday July 31, 2012 @12:11PM (#4317974)
Wouldn't compression between redis and hiredis make sense as well?

How much load would compression put on a Redis instance that talks to n clients where each client sends/receives medium sized JSON docs. And how much would be gained in terms of faster responses from Redis.

Re:Having fun with Redis Replication between Amazon and Rack... by stock_toaster on Tuesday July 31, 2012 @01:11PM (#4318389)
Something like lzf or snappy maybe?
Re:Having fun with Redis Replication between Amazon and Rack... by iand675 on Tuesday July 31, 2012 @01:59PM (#4318703)
In my experience, snappy with very small messages (< 200 bytes or so) can increase bandwidth usage, so I'd say the way in which one uses redis can impact if compression is useful or not.