News for hackers. Stuff that matters.
Re:Keeping Instagram up with over a million new users in twe... by mikeyk on Thursday April 05, 2012 @06:24PM (#3804956)
We have a variety of Redis machines, some of them are in a consistent hash ring (the ones we're using for caching); some are using modulo-based hashing (the ones where losing data on adding more machines isn't an option), and some are just single-node installs.
Re:Keeping Instagram up with over a million new users in twe... by cdelsolar on Thursday April 05, 2012 @07:21PM (#3805152)
How do you handle write replication? I haven't found any good document on how to do this (meaning, if the Redis master goes down, a slave should be promoted to master immediately).
Re:Keeping Instagram up with over a million new users in twe... by mikeyk on Friday April 06, 2012 @03:21PM (#3808497)
You can hypothetically use something like hearbeatd to do it; we run every Redis master with an attached slave and manually failover for now.

For a small team like ours, we prefer solutions that are easy to reason about and get back into a healthy state (it would take one server deploy to point all appservers at new Redis master), rather than fully automated failover and the "fun" split-brain issues that ensue. Of course that may change as we build out our Ops team, etc.