Re:The GitHub Availability Report
by user5994461 on Thursday July 16, 2020 @07:16AM
(#23857759)
>>> It's still unclear to me whether this particular failure mode is common enough
It is very common. Had the same thing happen in my company last year, on multiple systems! The organization was reaching a decade old, nobody thought of that when the table were initially created.
Sorry; it's hard to balance terseness with clarity. What I meant is to ask whether a tool such as Chaos-Mesh could have helped the github operations team to be aware of such a failure mode by causing this kinds of problems early.
I think that's the purpose of chaos-engineering right? Some things happen too infrequently for us to plan for them, until when they finally do, causing problems that are more costly to solve than the cost of the prevention.
It's still unclear to me whether this particular failure mode is common enough to be part of what a chaos-engineering tool explores by default; after all, not all databases around contain billions of rows. If engineers have to explicitly enable such a test because they are afraid of what would happen if a PK overflows, well, then probably they would take other preventive measures so that the problem doesn't happen in the first place (either preventively upgrading the size of the counter, or at least put some monitoring). I think what happened here is that a solution was designed when the database was way smaller; the user base grew and grew and this particular failure mode was overlooked (because there are many many other things to care about!).
(I write this because I'm genuinely curious about what other people think about this topic, not to bash on parent for a shameless plug; I mentioned that only as a side comment; I heard good things about TiDB, I wish you well)