Re:Twilio incident and Redis
by aidos on Wednesday July 24, 2013 @07:30PM
(#6099897)
As others have said, Redis is resilient in this configuration and by no mean ephemeral.
It's worth nothing from the original Twilio post "This cluster is configured with a single master and multiple slaves distributed across data-centers for resiliency in the event of a host or data-center failure". Ironically it may well be that not having the slaves at all could have prevented the outage.
I've seen other scenarios in the past where having a (mysql) master-slave arrangement has caused an outage. Root cause was bad configuration and a bug in the app code but had there been no slaves, there wouldn't have been an outage.
Just goes to show that it always pays to think carefully about the complexity you introduce into systems.
It's worth nothing from the original Twilio post "This cluster is configured with a single master and multiple slaves distributed across data-centers for resiliency in the event of a host or data-center failure". Ironically it may well be that not having the slaves at all could have prevented the outage.
I've seen other scenarios in the past where having a (mysql) master-slave arrangement has caused an outage. Root cause was bad configuration and a bug in the app code but had there been no slaves, there wouldn't have been an outage.
Just goes to show that it always pays to think carefully about the complexity you introduce into systems.