There are many MySQL tools which are able to automate the process of failing over between a master and a slave. Github initially chose a solution which would trigger this failover automatically, when a master was unresponsive for a particular amount of time. One bad query came in, and the tool did it's job by failing over from the master to the slave. Here's the problem - the query was automatically retried against the slave, causing it to fail over back to the master. Repeatedly.

The solution? Page when MySQL became unresponsive, let a human initiate the failover, and let the tools handle the failover itself (which is a very complex dance to perform manually). I think they eventually moved to a model where it would fail over once per few hours, and alert if the condition occurred again.