News for hackers. Stuff that matters.
Re:Addressing GitHub’s recent availability issues by vb-8448 on Tuesday May 16, 2023 @04:15PM (#35967159)
It's curious that all of the incidents are database related.
Re:Addressing GitHub’s recent availability issues by grepfru_it on Tuesday May 16, 2023 @06:36PM (#35968700)
No, well kinda. I think it's related to how github chunks data on the backend. That chunking methodology does depend on the database. They mention it worked elsewhere in the org, which makes me wonder if it was a fix to support their managed hosts.
Re:Addressing GitHub’s recent availability issues by fwlr on Tuesday May 16, 2023 @11:49PM (#35971051)
One thought: GitHub is basically a bunch of very complicated interconnected databases, no? I mean, to some extent all large web companies are, but GitHub much more than most. Any failure is going to be, in some sense, a database failure.

Another thought: this could speak to a critical weakness in GitHub’s database team. Maybe they hired a lot of bad developers to the database team at the same time, maybe a bad culture took root in the database wing’s management team.

Re:Addressing GitHub’s recent availability issues by vb-8448 on Wednesday May 17, 2023 @05:22AM (#35972941)
Looking at the issue details, I'll guess they have "poor ops practices":

- 9th may: during a deployment, rollback didn't work

- 10th may: a recently deployed function killed the database

- 11th may: failover didn't work

Re:Addressing GitHub’s recent availability issues by aaomidi on Thursday May 18, 2023 @11:09AM (#35989062)
Databases are legitimately the worst part of every company.

All the new, and much better tech is so fucking expensive.

CockroachDB has the worst sales people that can’t even demonstrate their own benchmarks.

Spanner is just unaffordable for most.

Postgres/MySQL are great but making them scale for write heavy workloads is a pain. I also hate managing primary/failover systems. We have consensus algorithms now. Let’s just use them.

Re:Addressing GitHub’s recent availability issues by Randis on Thursday May 18, 2023 @11:20AM (#35989228)
Care to expand on "CockroachDB has the worst sales people that can’t even demonstrate their own benchmarks"
Re:Addressing GitHub’s recent availability issues by aaomidi on Thursday May 18, 2023 @08:26PM (#35996274)
Ask the sales folk on cockroach to re-create their own benchmarks for you. They won’t be able to.