News for hackers. Stuff that matters.
Re:Days without GitHub incidents by otterley on Monday May 04, 2026 @02:16PM (#48012641)
Let me ask the question in reverse: what do you have against them such that the fellow human beings struggling to maintain their operations don’t deserve even a modicum of kindness, respect, and good will? Are you unable to separate the business from the hard working people behind it?

It’s not like they don’t know that people like us are counting on them: they recognize that their service is the “dial tone” for much of the world’s software development capability. They are keenly aware of the impact.

What happened to #hugops? Does it go out the window because those people happen to work for a company you don’t like?

Re:Days without GitHub incidents by StableAlkyne on Monday May 04, 2026 @02:32PM (#48012889)
When did OP blame the people involved personally?

If I to hire a contractor to redo my roof, and that roof leaks, whether they worked hard or not is immaterial. They did not do the task in they were paid to do. I'm not going to buy their services again just because their shingles guy was particularly charming.

MS has talented engineers, but that's a complete misdirection. Github is a service in decline: there is nothing wrong with criticizing them.

Re:Days without GitHub incidents by ebiester on Monday May 04, 2026 @02:37PM (#48012955)
I have all the empathy for people in the world.

A corporation is not a person. If your organization cannot handle the load, then you need to adjust your practices. The organization needs to prioritize their paying users. The organization needs to shift people from new features to keeping the lights on. And maybe the organization needs to find another strategy to manage its azure transition.

Re:Days without GitHub incidents by otterley on Monday May 04, 2026 @05:49PM (#48015484)
A corporation is made of people. GitHub cannot exist but for the people who continue to work for it. And they’ve already said, multiple times, that restoring availability is their top priority.
Re:Days without GitHub incidents by port11 on Tuesday May 05, 2026 @08:09AM (#48021356)
A corporation is made of people, but its ethos is the product of decision-making. If a corporation is consistently, say, unethical, is it because they hire only unethical individuals? Or because unethical people somewhere along the chain of command make unethical decisions?
Re:Days without GitHub incidents by otterley on Tuesday May 05, 2026 @03:02PM (#48026978)
I'm not exactly sure what you're getting at with this question. It seems to still conflate corporate-level decisions with boots-on-the-ground work.

Are you suggesting that whatever decisions their upper-level management makes that you consider unethical irreversibly and irrevocably taints all the difficult and honorable work that their engineers and operations people are performing?

Re:Days without GitHub incidents by port11 on Wednesday May 06, 2026 @02:38AM (#48033019)
I’m saying their lower-level employees are probably honest, hard-working people like everyone else. But the detachment that comes from a large corporate structure makes the higher-ups decide things that aren’t as honourable.

“Corporations are made up of people” is a strange way to excuse the reality that the ‘bad’ things that corporations do are often decided by top management.

Re:Days without GitHub incidents by otterley on Wednesday May 06, 2026 @09:36AM (#48036081)
Ah. I didn’t intend to excuse the decisions of upper management when I said that. My intent was to counter the notion that a corporation and its workers can’t be analyzed independently.

A corporation is just a business formation, and businesses are made of individual people working for it. Those people’s motivations and efforts can, and often should, be evaluated separately from the decisions of management.

Re:Days without GitHub incidents by port11 on Wednesday May 06, 2026 @02:27PM (#48039778)
We agree, thank you for the clarification. Have a nice day!
Re:Days without GitHub incidents by nimih on Monday May 04, 2026 @03:35PM (#48013842)
I think it's possible to be simultaneously: gracious and supportive towards the developers and ops staff who have been struggling to maintain reasonable uptime on the extremely important piece of shared internet infrastructure that everyone commenting probably relies on (either directly or indirectly) on a daily basis; and spiteful and cruel towards the massive (and, historically speaking, ethically fraught) corporation whose cynical acquisition and subsequent mismanagement of that same resource got us here in the first place.
Re:Days without GitHub incidents by otterley on Monday May 04, 2026 @03:46PM (#48013979)
I agree 100%! But this important distinction and nuance seems to be lost here.
Re:Days without GitHub incidents by ofjcihen on Monday May 04, 2026 @02:36PM (#48012944)
OP didn’t blame the staff. His focus is on the company.

Invoking individual workers well-being to defend a billion dollar company is also very strange.

Re:Days without GitHub incidents by otterley on Monday May 04, 2026 @05:48PM (#48015473)
A company is made of individual workers. That doesn’t change because there are a lot of them or that their employer has a lot of money.
Re:Days without GitHub incidents by ofjcihen on Monday May 04, 2026 @06:44PM (#48015943)
If anything your argument against MS’s uniqueness makes OPs case stronger.
Re:Days without GitHub incidents by otterley on Monday May 04, 2026 @07:00PM (#48016074)
Tell us how.
Re:Days without GitHub incidents by turtlebits on Monday May 04, 2026 @02:30PM (#48012853)
#hugops is to your coworkers, not to the nameless big-corps who can't maintain a service for paying customers. You should be raising a shitstorm when things you pay for aren't reliable or unusable.

Hot take, if it's traffic is causing issues, throttle your free-tier, pause signups, or stop giving out free things (like runner time).

Re:Days without GitHub incidents by otterley on Monday May 04, 2026 @05:50PM (#48015492)
Who is “maintain[ing] the service”? The workers, of course!
Re:Days without GitHub incidents by logicchains on Monday May 04, 2026 @02:22PM (#48012731)
>What happened to #hugops? Does it go out the window because those people happen to work for a company you don’t like?

Would you feel the same way about a colleague who kept causing downtime in your product again and again, seemingly without making any progress in addressing whatever issue was causing their repeated mistakes?

There are web applications out there that are far more complex than GitHub but have much less downtime. It's not like they're facing an unsolvable problem.

Re:Days without GitHub incidents by otterley on Monday May 04, 2026 @02:27PM (#48012810)
You don’t know that it was “their mistake.” Unless you’ve personally successfully scaled a suite of nontrivial services equivalent to GitHub’s to accommodate an unexpected 14x increase in traffic, you respectfully have no basis for such an assertion.
Re:Days without GitHub incidents by dijit on Monday May 04, 2026 @02:37PM (#48012961)
I have.

You could argue the scales are different, but computers are also faster now.

So, argument to credentialism out of the way... What should we do as consumers if a provider that is a defacto monopoly due to network effects stops functioning?

https://news.ycombinator.com/item?id=47947719

https://www.linkedin.com/in/jharasym/

Re:Days without GitHub incidents by otterley on Monday May 04, 2026 @02:52PM (#48013193)
> You could argue the scales are different, but computers are also faster now.

Scale is everything and a faster computer doesn’t always help. Vertical scaling has limits, and complex distributed systems are complex.

Since you seem to possess a diagnosis and remedy with a reasonable amount of certainty, I’m sure they’d love to hear from you and have you fix all their problems for them. Especially if you can do it while not making the problem worse in any dimension.

Re:Days without GitHub incidents by dijit on Monday May 04, 2026 @02:58PM (#48013287)
The link in my previous comment answers the credentials question in detail- including specific technical post-mortems on horizontally scaled stateful systems. Vertical scaling wasn't the topic.
Re:Days without GitHub incidents by otterley on Monday May 04, 2026 @03:06PM (#48013412)
You’re missing the point: a doctor doesn’t diagnose and practice medicine on a patient he hasn’t thoroughly evaluated himself. This is the sort of wisdom that a staff engineer and CTO is expected to have earned.
Re:Days without GitHub incidents by Aurornis on Monday May 04, 2026 @03:02PM (#48013364)
> I have.

I skimmed your profile. Working on the infrastructure for a couple mid-tier video games is a cool accomplishment, but equating this to having solved GitHub level scale rings hollow.

GitHub has a couple orders of magnitude more daily active visitors than the games you worked on had at their peak.

You can make valid criticisms of GitHub without trying to reduce their scale or inflate your credentials to create a false equivalence.

Re:Days without GitHub incidents by dijit on Monday May 04, 2026 @03:21PM (#48013654)
"false equivalence" needs an equivalence claim to be false.

I didn't make one. The sentence after "I have" was literally "you could argue the scales are different."

GitHub spent a decade asking the world to host its code with them. They got what they asked for. You don't get to beg everyone to run services for you for ten years and then have "scaling is hard" be the answer. They should be improving, not regressing over time, and they have some of the worlds best engineers and a trillion dollar corporation behind them, they don't need my sympathy.

The original question is still open and nobody's engaging with it.

Re:Days without GitHub incidents by Aurornis on Monday May 04, 2026 @07:24PM (#48016270)
> I didn't make one. The sentence after "I have" was literally "you could argue the scales are different."

Don't you at least see how it's misleading to respond "I have" in response to a question about scaling GitHub-scale services?

Trying to caveat it with "the scales are different" misses the point. The parent commenter was talking about scale.

Re:Days without GitHub incidents by cnewey on Monday May 04, 2026 @03:18PM (#48013615)
I'm not sure that resorting to personal attacks against the parent commenter for making a legitimate critique is the right, fair, sensible, or mature approach here.

Discarding legitimate criticism based on some self-determined criteria of intellectual superiority isn't a good look. It smacks of elitism and isn't something conducive to a productive and positive community discussion.

It is unhelpful, rude, condescending, and completely fails to address the underlying problem.

Re:Days without GitHub incidents by otterley on Monday May 04, 2026 @05:44PM (#48015430)
The commenter inserted his own personal bona fides (as a proxy for skill, experience, and knowledge) and use them to bolster his conclusion of culpability and incompetence of the GitHub team. If you take that risk, you should expect to be challenged if those skills are not up to par.

Put more simply: if you get into the ring, you’d better be prepared to take a punch.

Re:Days without GitHub incidents by Aurornis on Monday May 04, 2026 @07:18PM (#48016219)
Not a personal attack to fact check someone's claims.

I didn't bring their credentials into the conversation. They did.

Re:Days without GitHub incidents by VirusNewbie on Monday May 04, 2026 @11:36PM (#48017785)
Ok, well, I work on systems quite a bit larger than Github, and I think they have a major reliability issue.
Re:Days without GitHub incidents by otterley on Tuesday May 05, 2026 @01:11AM (#48018301)
That’s not in dispute. The question is whether we should be supportive of the company’s efforts to improve reliability, or whether we should keep punching down. How would you feel if you were in a similar situation and outsiders breathlessly provided uninformed opinions about your problem and questioned your competence?

It’s all about the Golden Rule.

Re:Days without GitHub incidents by mattmanser on Monday May 04, 2026 @02:36PM (#48012945)
Yeah, they should be testing for that, right? I think there's a lot of people reading comments like yours and thinking, is this person a paid shill or what?

The earn bucket loads of money, they should be planning for exactly that. And testing for it via load testing every day.

Perhaps you've forgotten the days of GitHub presenting themselves of software engineering thought leaders.

Re:Days without GitHub incidents by otterley on Monday May 04, 2026 @02:49PM (#48013156)
I’ve worked at some very well-endowed organizations. Having money is no guarantee of a particular outcome. There is a lot of money chasing a limited supply of talent. Moreover, distributed systems that were built long ago with certain assumptions can’t be refactored as quickly as the HN populace might believe. The Mythical Man-Month is a popular book for a reason.
Re:Days without GitHub incidents by c-hendricks on Monday May 04, 2026 @02:48PM (#48013142)
> Perhaps you've forgotten the days of GitHub presenting themselves of software engineering thought leaders

Genuinely could use a refresher here.

Re:Days without GitHub incidents by VirusNewbie on Monday May 04, 2026 @02:22PM (#48012732)
Executives have made a choice to not pay for top talent at Microsoft Azure and Github.
Re:Days without GitHub incidents by otterley on Monday May 04, 2026 @02:29PM (#48012850)
Would you consider telling this to the people working at GitHub directly? I’m sure they’d appreciate your evaluation of their skills and talent.
Re:Days without GitHub incidents by falcor84 on Monday May 04, 2026 @02:34PM (#48012914)
There are two options, either they are lousy at their jobs, or they are incapable of pushing back against unrealistic demands. Neither is a good indicator of their skill and talent as engineers.

I know I am speaking from a position of some privilege, but I have previously left workplaces that did not allow me to practice good engineering, and I do expect others to do so.

Re:Days without GitHub incidents by vablings on Monday May 04, 2026 @02:44PM (#48013086)
In a SWE job market like this, do you really want to be seen as the "conscientious objector"?

There are literally thousands of people who are ready to ride up the totem pole, it would not be a difficult decision for a bad manager to swing his axe and replace the new head

Re:Days without GitHub incidents by einsteinx2 on Monday May 04, 2026 @03:03PM (#48013373)
Talented engineers shouldn’t have much problem finding another position even in this market (of course they should find one before leaving I’m not discounting family responsibilities and whatnot), so if your argument is they’re not able to leave and find another job then you’re essentially agreeing with the person you’re replying to.
Re:Days without GitHub incidents by fragmede on Monday May 04, 2026 @03:05PM (#48013403)
Or, they've been given crap primitives to work with. There's only so much lipstick you can put on a pig. I don't know what database they're using or what their pub sub and streaming looks like, or even what their system diagram actually looks like. But, well, you don't see Google having these kinds of problems. Other ones, sure, but between Chubby and Spanner, if Google had bought GitHub we wouldn't be having these problems.
Re:Days without GitHub incidents by falcor84 on Monday May 04, 2026 @07:20PM (#48016233)
But it wasn't a pig. It was a reliable system, and then it increasingly became an unreliable one, in a way that is not explainable by the mere increase in demand. Whatever rearchitecture was performed, it was done and is apparently being perpetuated by software engineers who should be held accountable. Not necessarily guilty, or even directly at fault, but accountable nevertheless. "I am just an employee of a bad company" is not a valid excuse for an engineer.
Re:Days without GitHub incidents by VirusNewbie on Monday May 04, 2026 @11:34PM (#48017780)
eh...

https://github.blog/news-insights/company-news/oct21-post-in...

They tried to scale MySql and turn it into Cassandra and then they lost customer data (that they claimed to later recover).

Re:Days without GitHub incidents by otterley on Monday May 04, 2026 @05:51PM (#48015495)
Really? Only two possibilities?
Re:Days without GitHub incidents by VirusNewbie on Monday May 04, 2026 @02:40PM (#48013018)
yes I would tell them "you're underpaid, if you can, come to a company that appreciates your talents more".
Re:Days without GitHub incidents by otterley on Monday May 04, 2026 @02:46PM (#48013106)
What you said came across as an adverse judgment of their skill and talent. Is that not what you meant?
Re:Days without GitHub incidents by VirusNewbie on Wednesday May 06, 2026 @12:45PM (#48038358)
I've interviewed at many large tech companies including Microsoft, and Microsoft Azure was mostly a clown show. So, yes I judge the talent there. I'm sure there are some superstars that I'd love to work with, maybe I'd even work there and try to fix it for the right price, but god damn was it a stark contrast to other companies.
Re:Days without GitHub incidents by tardedmeme on Monday May 04, 2026 @02:50PM (#48013166)
Are you hiring?
Re:Days without GitHub incidents by VirusNewbie on Monday May 04, 2026 @03:03PM (#48013369)
Yes google cloud is actively hiring.
Re:Days without GitHub incidents by estimator7292 on Monday May 04, 2026 @03:15PM (#48013575)
If you pay someone full price to do a job, they know they can't fulfil the terms up front, accept the work, deliver less than the agreed upon terms and still charge you full price, you'd probably call that transaction fraudulent.

GitHub is promising service they know they cannot meet, not telling you that, and still charging you full price. What's more, one can argue quite convincingly that they're lying about their level of delivered service by not reflecting the actual level of uptime on their status page.

To give benefit of the doubt requires that the other party is not blatantly and overtly acting in bad faith. When they are, you're just apologizing for fraudulent behavior.

Re:Days without GitHub incidents by otterley on Monday May 04, 2026 @03:49PM (#48014012)
Fraud is a serious civil and criminal accusation that’s not to be taken or given lightly. Can you detail the fraud that’s being committed? What is the specific promise they made that you’re being deprived of? Remember, the four corners of your agreement with them are controlling.