edjboston

Thank you.

It's not quite over though. Monday traffic will be far higher than the weekend. People will need to login, which will put unusual stress on those endpoints.

Also, there is a small chance someone experiences malformed or missing data. There was only a single repo that we know experienced this, which we're already working on. So I'm only talking about unknown unknowns. So we will be on high alert to respond to any such requests and retrieve data from the archives in the previous infrastructure.

GitLab VPE here. This instinct proved correct. We'll look into editing the video this week after the team gets a breather.

GitLab VPE here. We're working on Dockerized version that will run in a Kubernetes cluster, which should make this easier, assuming you've already made that investment in your infrastructure.

More pertinent to your concern about cost: It should make it easier for PaaS' who are k8s-based to run it for you and perhaps provide a lower price point.

GitLab VPE here. We are starting to re-write performance intensive part of our application in Golang. E.g. our Gitaly project: https://gitlab.com/gitlab-org/gitaly

GitLab VPE here.

Historically the dominant source of outage minutes is indeed features that didn't scale (70%).

However, We've made great strides in the past 6 months on QA and release management and it's yielded a marked improvement in availability. The last week has been an exception to that.

We're in the midst of a move from Azure to GCP, and once that is done, we're going to rebuild out system to be entirely automated which will eliminate a class of manual mistakes.

GitLab VPE here. We apologize for the site performance degradation today.

We do have a dedicated DB team of three people. We also have an open vacancy for the team's manager and and individual contributor.

We are in the midst of a move from Azure to GCP. That means more work than usual is going on currently because we're replicating data across infrastructure-as-a-service providers and keeping the site running. This has been the case for a couple months now and it's shouldn't impact site performance. Today's site slowness was unfortunately due to a manual mistake related to a Postgres upgrade task.

[2] This article is old. A move to metal never happened, and is not in our future plans. In fact, after we move to GCP we are likely to set up a CloudSQL replica and evaluate it's performance

Gitlabber here.

We have a plan to address this and are kicking off our move to GCP shortly. Our top-level engineering department goal is to make gitlab.com ready for mission critical workloads and are targeting industry standard SLA's (e.g. three 9's). You can see our proposed architecture here if you're curious: https://about.gitlab.com/handbook/infrastructure/production-...

Another GitLabber here.

We just kicked off a major effort to address the availability issue Sid mentions above. The highlights are that we're moving to GCP which should provide better underlying reliability. But interestingly we found that only about ~20% of our downtime minutes where from underlying infrastructure. Whereas ~70% came from features that didn't scale.

So the more exciting part of the project is to tighten the feedback loop between development and deployment with a continuous delivery pipeline. This may be obvious to some people, but it's harder to pull off when you've got an open source project, an on-prem product, and a large-scale SaaS sharing the same code base. I'm calling it "Open-core SaaS" and there are only a handful of companies that run a large, multi-tenant service based on an open source project.

My company Airware (YC W13) is hiring several Geospatial Engineering positions to work on our drone data platform.

Apply http://grnh.se/yn3epi

Excuse the self promotion, but my company Airware (YC W13) is hiring several Geospatial Engineering positions to work on our drone data platform.

Apply http://grnh.se/yn3epi

Check out our Eng blog https://makers.airware.com

Excuse the self promotion, but if you're interested also check out a UI Docker/K8s deployment tool my team open-sourced earlier in the week. It's opinionated, but it might meet some of your needs:

https://makers.airware.com/open-source/vili/

Can you say more? If I understand what you mean by prototype and refactor it seems like a reasonable way to iterate.

What I was getting at in the article was completely transferring a production app (presumably under load and with customers) from one language to another.

This is awesome. Wouldn't it be cool is this were wired up to an orchestration tool like Terraform to visualize TF states?

Another level would be container infrastructure. Our new project is Docker/Kubernetes so the AWS infrastructure mostly just vanilla resources. It would be fantastic to have a similar tool for the internals of our micro SOA.

Great job!