My company uses the same DB sharding tech. It took us about a year of daily / weekly outages until we finally were able to fix our performance issues. 256 database splits, lots of cross-shard queries removed, etc before we finally reached a happy-ish state for a year.

Now it’s scheduled to fall over in about 6 months and everyone is freaking out again. It’s not new features that are hurting us, it’s the existing core product line. All the new stuff is built on horizontally sharded DBs from the get-go.

Wow. What kind of tech debt are you reckoning with / what better design decisions would you implement if redesigning today?

Used to work at BigBank$ where they had an internal 8 node Bit bucket cluster. Down multiple times a week for 6+ months before they could finally get it under control (Jira + CI was apparently putting a lot of load on and apparently BitBucket doesn't scale past 8 nodes)

Needless to say, self hosting can be more reliable but you'll probably end up dedicating a lot of resources to building that out and supporting it

Single VM (backed up) with gitlab, k8s for workers.

Not difficult, not hard to maintain. Worked fine for our ~500 projects (each with pipelines/ci|cd) and ~100 devs.

I've never tried to self-host something like that, but on the other hand, I've never worked at a place where the internal GitLab wasn't a shaky mess, particularly when CI or Pages or certificates is involved.

I believe people when they tell me they've managed it without major problems, but clearly they're the 10x developers of the self-hosted world, because the people I work with seem to find it hard - and that is not a slight on them, because clearly even Github can't do so much better!

Great talk on this topic from 2015 by the team from Netflix: https://www.youtube.com/watch?v=aWcKsX0_qAg

How are you doing cost analysis between moving off of GitHub and working around the issues?