Blaming software/database complexity as an excuse why GitHub hasn't done so is extremely pathetic. Systems almost always gets more complex over time, not less.

Until you've done it yourself successfully at GitHub's scale, I think you should withhold judgment.

There's a lot of people on HN, including myself, that have scaled MySQL beyond what we thought was possible.

The hard part isn't solving the scaling problem, but the uncomfortable conversations with leadership about how chunks of roadmap will be drastically delayed. Inevitably, leadership makes a decision to kick the can down the road, further exacerbating the problem.

why do you blindly assume the person you are replying to has not worked at scale?

hundreds of engineers have worked deeply on sharded mysql at massive scale, many of us comment on HN!

My experience is that people who know how challenging big, complex changes can be know better than to trivialize their difficulty when discussing them.

so i would agree when talking about comments like "i could code twitter in a weekend! why so many engineers there!"

but this is different. github is a 14 year old company, with annual revenue in the hundreds of millions USD

I literally cannot think of any other comparable size and age mysql user who has not successfully sharded long ago and avoided outages of this magnitude. and in return we get hand-wavy excuses of "complexity!!!" from their former vice president of engineering who was previously also their first DBA.

criticizing these excuses is not trivializing the complexity, it's more of a "we all did this at our respective companies, who also had a lot of complexity! why can't you? we are github users, we rely on you and are very unhappy, we want to know how this happened" and just getting excuses as an answer.

> I literally cannot think of any other comparable size and age mysql user who has not successfully sharded long ago and avoided outages of this magnitude.

How about a little humility? You have no idea who else has similar problems out there. I'm sure Percona and other DB consulting companies would tell you otherwise, as would PlanetScale. (If this weren't true, they wouldn't be viable businesses.)

> github is a 14 year old company, with annual revenue in the hundreds of millions USD

You're making the classic mistake of thinking that having money means you can just wave your hand and hire whoever you want with whatever talent you need to solve your problems. The world just doesn't work that way.

As for the "hand-wavy" excuses, well, the people who know the issues don't need to disclose the dirty details in a public forum, nor are they often able to because of NDAs and other legal encumbrances. And it can be a career-limiting move to throw your colleagues under the bus.

You can choose to assume people are incompetent, or instead choose to assume that people are working as best they can under the constraints placed on them. I think it's better to assume the latter.

I do in fact know who else would have had similar problems, directly from personal experience and experience of coworkers and friends. large-scale mysql community is fairly insular, lots of talent rotation between the large scale companies and between percona and pythian, everyone knows everyone else, everyone talks war stories at the conference bar and elsewhere. but thanks for making assumptions once again!

i am NOT saying that github can hand wave this away with money and hiring. you misunderstand. what i am saying is company of their business size and scale should have already handled this long ago, because that's what literally every other large comparable company has done.

there are front-page-of-HN threads about multi-hour github outages every single day this week. show me an example of another similar-size company having an equivalent meltdown please. only real equivalent is twitter during fail-whale days when they were only a few years old. they solved it early on, as should have been done.

i am not saying the entirety or even majority of github is incompetent, but i am saying there is clearly something extremely wrong and extremely unusual that led to this, and citing "complexity!" is just a pile of BS.

You appear to know some people and organizations who have solved the problem. What I am saying is that, other than GitHub, you apparently don't know the multitude of other organizations that haven't solved the problem yet, and you appear to falsely believe that they do not exist. (And yet you also claim to know people from Percona and Pythian; what do they tell you they do for a living?)

As I said, a lot of the problems organizations have are not publicly disclosed, and the people who know aren't authorized to talk about it. So I'm sorry, you're not going to get the examples you seek here. Companies like to keep their weaknesses close to their chest.

> i am NOT saying that github can hand wave this away with money and hiring. you misunderstand. what i am saying is company of their business size and scale should have already handled this long ago,

If not through money and hiring, how should they have accomplished it? You propose a goal but no actionable plan. That's worth diddly squat, both in engineering and in business.

i have personally been on teams that solved this problem and worked on this problem. across multiple companies. with larger DB size than github's.

if you think Percona and Pythian implement sharding solutions, you are deeply mistaken, this is not what they focus on at all. advice, sure. more on the perf and ops side. but not sharding implementation, and definitely not application side of things.

furthermore i am saying if other major companies were having this type of issue, everyone in the public would know about it! because the product/site/company would be down all the time. this isn't some thing you can hush hush. close to the chest? how do you keep daily outages close to the chest? completely absurd

there is literally no analog in the US tech world to a large 14 year old company having daily outages for a week, preceded by multiple outages per month consistently for the past year+.

i will stop replying now because it is clear we live in different realities or smth

Somehow you are very confident that this github issue can be avoided by horizontal sharding? I am not so sure about that. If my hypothesis is right (see my other comment in this HN discussion), then it can be rather trivial to bring down any MySQL instance or shard, such that the end result would be similar regardless of horizontal sharding.

I would also back otterley's position that there exists many other companies with larger non-sharded MySQL instances than github's. They may or may not have better reliability than github.

innodb global lock architecture has improved greatly since 5.1. ancient bug report from harrison is not indicative of problem here. what you think, facebook has unsolvable database cpu pileups for 12 years straight? harrison would not have been promoted to vice president if this was acceptable in his shop

"They probably outsource the lock queuing logic to their Memcached layer." you are grasping at straws. the wrong straws. go attend some facebook conf talks, meet their engineers, read some arch papers, and stop speculating this nonsense. largest db tier there does not even use memcached, hasn't for 9 years. read the TAO paper!

horizontal sharding works great for many many companies with databases several orders of magnitude larger than github, why would it not work for github?

you are also saying there may be large companies with lower reliability than github? ok name them. again, if this was the case, everyone would know! their availability at this point is like what, 1 nine?

> innodb global lock architecture has improved greatly since 5.1. ancient bug report from harrison is not indicative of problem here.

No, this performance cliff does still exist in the current mysql 8.0 branch. The Contention-Aware Transaction Scheduling added to 8.0 doesn't help the queuing problem with hot row at all. So facebook either doesn't suffer from the problem, e.g. all their writes are append-only so there is no hot row, or prevents the problem from happening, e.g. with API rate limiting built on top of something like memcached.

The improvement in innodb lock architecture mostly comes from splitting the locks into more granular levels. However, updates to the same hot row would require the same lock at the page-level. It is already as granular as it can be. So the tag of the bug report is correct, this is a problem of "lock queuing".

Not every organization that is either behind the 8-ball on their MySQL scalability projects or who is suffering incidents as a result is making headlines. That’s just a fact, and you’re just going to have to accept that you don’t know everything.

And frankly, given your attitude on this, I think you’re going to need to present some bona fides for anyone to take you seriously. Maybe you can even convince GitHub to hire you and solve their problems. I bet they’d be happy to listen to you berate them on how pathetic they are.

i was not even the commenter who said they were "pathetic", sheesh now you are attributing others' comments to me!

i have posted many correct technical details in various subthreads of this post. i don't care if you believe me, i do not live in your reality where constant downtime is totally fine and major companies are down all the time but no one knows about it

There have to be thousands (if not more) people that have. Why assume that they have not?

This is a super, super annoying type of comment, and against HN guidelines. It adds absolutely nothing to the conversation. Don't go digging through people's comment history to throw mud.

i did not dig through his comment history. i was directly participating in that thread i quoted! and it was a recent thread. please check it again.

i agree though my comment could have been nicer. apologies.

as to adding nothing to the thread: disagree. he literally dumped on AWS with false claims. how do you know i don't work at AWS? why is it ok for him to say stuff like that?

Let these types live in their supreme know-it-all fantasy world, they're not worth a thought. Anyone with a little common sense will see through their shenannigans and proceed to ignore them the way you'd ignore an ant on the sidewalk as you go about your busy day.

Don't take it personally, their ignorance has nothing to do with you.

The issue isn't just a lack of "niceness" but also how this type of comment derails the conversation into an offtopic argument which should have been pursued only in the original context.

If you disagree with the particular assertions here (that your are grossly oversimplifing without context), you should address it directly. Saying "it's fine because you do it too" isn't so much a lack of "niceness", but rather how this type of comment unproductively derails the conversation into an offtopic argument while adding zero constructive discussion.

> offtopic argument which should have been pursued only in the original context

I would have gladly had that discussion in the original context! but he did not reply to me there, nor to any of the other 6 people who replied to that horrendously incorrect comment of his

i would have let it go, but yet here he calls me out saying my own comment is grossly oversimplified, but he offered little explanation of his perspective. if i am understanding his brief argument it is:

1) github is old.

2) sharding is hard.

ok so here is my response: it was a lot less old when he started working there in 2013. many other companies have sharded several years after launch. yes it is hard. i have directly worked on teams sharding larger DB's than github's, sharding legacy apps. and so forth. it can be done.

it is really hard to justify github to be down this many times from the same root cause. year after year

> I would have gladly had that discussion in the original context! but he did not reply to me there, nor to any of the other 6 people who replied to that horrendously incorrect comment of his

I think your response in that thread was sufficient. If you feel it was insufficient, that thread would be the correct place to continue the discussion or further clarify any misinformation where people may actually it.

People rarely admit when they are wrong and you don't have a right to (or even a reasonble expectation of) any such admission in this context. Harassing them about it in other threads is not a positive contribution to the discussion or the community.

he accused me of making a gross simplification. i responded, perhaps unkindly. but at no point have i "harassed" him.

github's outages have made my life difficult. planetscale's pricing scheme has also burned me. i am not just attacking this guy at random.

based on upvote points i can conclude many people do think i am contributing positively to this discussion. please refrain from tone-policing me. i don't see any technical comments from you in this discussion whatsoever so how exactly are your comments positively contributing?

I already explained that the issues was not your tone, but your attempt to revive an argument from a different, unrelated thread, in this one.

i did so because it is directly relevant to this topic!

the person who for many years was directly responsible for github's databases, and later their overall infrastructure, recently made a horrendously inaccurate statement about the industry's most popular managed database offering

this shows either a severe lack of technical understanding, or a willingness to make deliberate technical misstatements

now github is down literally every single day due to database problems. it is very relevant! accountability is important!