Re:Twilio incident and Redis
by marshray on Wednesday July 24, 2013 @10:20PM
(#6100553)
Regardlesss, something about the redis query returning a zero balance was triggering bad behavior in the billing system.
Also, reading between the lines in the incident report made it sound as if there might have been multiple teams involved in the troubleshooting and not communicating perfectly. For example, were the redis admins informed that customers had been getting billed repeatedly when the decision was made to restart the billing system? Did they have access to the billing system logs which might have contained errors related to redis being read-only?
All in all, big props to Twilio for starting to get customer accounts credited back within 11 hours of the first trouble and even more for their wonderful open disclosure.
Also, reading between the lines in the incident report made it sound as if there might have been multiple teams involved in the troubleshooting and not communicating perfectly. For example, were the redis admins informed that customers had been getting billed repeatedly when the decision was made to restart the billing system? Did they have access to the billing system logs which might have contained errors related to redis being read-only?
All in all, big props to Twilio for starting to get customer accounts credited back within 11 hours of the first trouble and even more for their wonderful open disclosure.