Re:Airbnb Tech Talk: Jeremy Edberg of reddit and Netflix -- ...
by redthrowaway on Monday October 22, 2012 @01:29AM
(#4681972)
As an aside brought on by the fraternal ballyhoo going on here, do the ex-reddit guys still keep in touch? It's kind of sad that Eric (and, I suppose, Alexis) are the only members of the old guard still kicking around there.
It got progressively worse as time went on the EBS performance got worse. It was happening every couple of weeks after you left, and I think it got even worse after that.
Luckily we had dug into the londiste internals and managed to figure out how to restore replication much faster, but that was still super painful. :)
But you're right, the disk lag usually caused the immediate problem -- the out of sync slave was usually pretty easy to deal with, depending on how many got out of sync at once. It was just annoying and time consuming.