And here we are with our on prem Gitlab with CI/CD that hasn't been down for 2+ years (except for very short, planned maintenance) and has 100% predictable cost at much lower rates than cloud services. We can still scale within minutes if needed (thanks to Terraform).

Dedicated servers are just so cheap, esp with sth like CI/CD that's perfect to host at cheaper providers.

Seeing constant issues with Gitlab and Github products just reinforces that decision.

That isn't everyone's experience. Our onprem Gitlab CI/CD sometimes breaks, usually because the Docker daemon went down for unclear reasons. The fix is easy, just restart the daemon, but still. Even managing a server may not be a trivial task depending on the available competency in the organization.

Upgrades and backups are way more problematic. A while ago I discovered that backups broke and I had to fix it. I setup automatic APT upgrades for Gitlab, but every few major versions or so a manual upgrade process is required. In an organization where people are less vigilant, and/or haven't setup monitoring of backups, this can easily lead to a disaster down the road.

None of this is particularly hard — for me. But for some others this may as well be black magic. Furthermore all this work is a nuisance, a distraction from what I usually do.

Your mention of APT suggests to me that you're trying to maintain a stable deployment on a pet server, whereas GP's mention of Terraform suggests they've built some sort of pipeline for repeatable deployment.

GP's approach does have a bigger learning curve for those that have never worked that way, but after that the ongoing maintenance is simpler, less time consuming and much lower risk. That probably explains most of the difference in your experiences.

We don't have an overly complex setup, just some automations along the way to make things easier the long way. Sure, it's a bit more complicated than manually installing a server, but learning how to use terraform and automating patching/backups etc isn't rocket science.

Anyone with some knowledge of Linux/Programming shouldn't struggle setting this up within a few days. And it adds knowledge to the company that's very valuable.

With Gitlab, as with most software, restarting it regularly outside of office hours is the best way to ensure a stable service. Also something SaaS can't do, as they never know when their customers need the software.

Many companies with on prem setup also don’t know when their devs need the software. Some have people in multiple timezones, some do async work, etc.

Yeah but how bad is a 5 min restart for that one dude in Australia at odd hours?

How bad is a github downtime? I guess it all depends.

Our deployment is repeatable. Pet server but managed with Ansible. If it's on fire, provision new server, rerun Ansible, restore backup. Server manually provisioned instead of Terraform, but provisioning the server isn't the hard part anyway.

If that's your setup, we also just have Pet servers. Sounds pretty professional to me.

I bet the GitHub/Lab/Atlassian/etc customers affected by the recent outages would be delighted to be able to SSH in and fix it or at least manually extract any required data until the outage is properly resolved.

It sounds as though you could probably learn a thing or two from how the GP has set things up if their experience is so much different from yours.

I actually never encountered the problem with the crashing daemon, but we also have periodic restarts (they don't take too long and can be scheduled to happen at night).

Upgrades aren't automated but as the server is only exposed to the internal network (and also not fully exposed there), the attack surface is much lower. Keeping servers up to date isn't hard, and we don't always insist on running the newest Gitlab version. We make sure to upgrade if there's a very critical bug or if we lag too far behind.

It's true, upgrades for Gitlab could be smoother, but we've never had significant down time from it. And while others might say it destracts from day to day work, I (and some others) find it a welcome break from coding.

"And while others might say it destracts from day to day work, I (and some others) find it a welcome break from coding"

Taking a actual break works too, I think. I mean, it is a virtue to see good in everything ..

> automatic APT upgrades

Please, please don't ever do this. Or if you do, make sure you're taking a snapshot of the box every hour or something..

If you don't have a dedicated sysadmin who's always on top of new Gitlab versions, then the alternative to automatic upgrades is letting insecure versions run for an arbitrary time until someone takes the time to upgrade. Gitlab is merely 1 of the 200 things we have to keep track of.

Besides, we already make weekly automatic backups, stored remotely and encrypted. And Gitlab automatically makes a locally-stored backup before each upgrade, which is good enough unless the disk fails right after we discover that the upgrade is broken.

You can configure just security patches and not full upgrades.

What about automatic security APT updates?

Or just before the upgrade?

That's how we do it— snapshot the box, apt dist-upgrade, snapshot again, gitlab upgrade, snapshot again.

A few days later, drop the snapshots.

> Furthermore all this work is a nuisance, a distraction from what I usually do.

We said the same. Now we lose twice as much time due to Gitlab.com outages

This seems like a Docker issue more than a GitLab issue. We run ours in GKE, so the infrastructure is managed and auto scales.

We switched from self-hosted Gitlab to Gitlab.com in the name of reducing infra burden...my oh my do I regret that choice. Same experience to you - no downtime that wasn't planned when we self-hosted. Load was very predictable, and ran on k8s anyway. Much, much faster than the hosted version.

And I don't get what infra burden there is. We run Gitlab, Mattermost, Databases etc and running/maintaining those is still just a very minor part in the work of some developers (who like and are good at infra). Put everything behind Tailscale/Cloudflare Tunnel and your attack surface is very low to start with. Most deployment and patching can be automated anyway, so there's not much left. In the end we save a lot of cost, avoid lock-in and keep much more knowledge in-house.

It’s just one more thing to backup, test, monitor, etc.

Upgrade migrations didn’t always succeed so then you need to go back and remember what you did last time to fix it.

It’s a cognitive burden and distraction. In theory it’s cheaper to pay Gitlab to handle for us. Except that now we lose more time due to Gitlab.com outages than we spent maintaining on prem.

That's exactly it, it adds a list of nodes to the "required knowledge" graph of your company, adding job positions, expenses, people, and (operational?) risk to your company that I'm sure already has its own list of things to worry about.

There's so many companies out there whose infrastructure is run by this One Guy. Any company needs to assess whether they can risk losing their One Guy (see also: Bus Factor), or if they'll just have to swallow the cost and (some) risk of (some) downtime by using a service provider.

But is it much different than the usual CI or infrastructure house of cards/YAML files?

Every company has their own way of doing cloud-based infra, CI build pipelines (in their favorite CI provider's dialect, since those aren't portable either), etc.

Every time I start at a new place I have to reverse-engineer what the previous guys did; there is no standard. By that benchmark, one extra thing to document/keep in mind isn't that big of a deal, especially considering GitLab (installed using their standard "omnibus" install on one of their supported distros) is likely to be more standard than all those custom Terraform/CI pipelines/etc.

Any decent developer should be able to install and maintain on-prem Gitlab. It's not harder than running most managed services on AWS (I'd say it's simpler). So you already have the people, you just need a few minutes (literaly minutes, not hours) of their time per week/month.

There's operational risk, but that doesn't go away if you shift to Gitlab.com infra. Just in this case, you can control all of it.

It's really not this simple. Even if it were just minutes (it isn't, but agree it's not huge overhead) you still have the operational risk when things inevitably go wrong. Now you're impacting your orgs ability to be productive (super expensive) and you incur the direct cost of fixing. Who do you think is going to be better at triaging Gitlab: someone looking after on-prem once a month, or Gitlab's team that do this full time every day. That's why people pay for SaaS.

The problem with Github and Gitlab is that it's actually turning out that those companies aren't that good at running their own product, which is tilting the economics of the above.

> someone looking after on-prem once a month, or Gitlab's team that do this full time every day

Keep in mind that Gitlab.com sees much more load & activity than your on-prem equivalent, and their maintenance schedules don't line up with yours. Ideally, you don't want someone messing with your VCS/CI "every day".

Your on-prem is not only very unlikely to need any of the moving parts that a world-scalable system needs, but any maintenance can also be planned out-of-hours to make sure eventual operator errors don't cause downtime during business hours.

In theory, yes, but in practice most of the difficulty in maintaining an on-prem Gitlab will be that individual company's bespoke installation of it.

> Any decent developer should be able to install and maintain on-prem Gitlab

By this measure, a lot of companies have zero decent developers.

I might be skewed because I do a lot of DevOps, but this is clearly one of the easiest thing to do for any half decent dev. You juste have to follow one half hour tutorial (and have a very basic knowledge of VLANs, DNS and routing, maybe? Depending on your internal network). The hard part is making sure your downtimes are low or inexistants, and if you this, hire a devops.

Backups tend to take the most work, especially if you verify them periodically. But at least that means you have backups. I'm not sure I would want Gitlab to have all my data without a local copy for such important information.

An issue everyone seems to overlook is that an SaaS typically needs to be scalable to the entire world while your on-prem service pretty much never does.

The SaaS will have more moving parts & operator activity which in turn makes outages more likely, and that "operator activity" can't be planned for unlike on-prem where you can schedule maintenance out-of-hours to make sure any accidents don't affect the business.

Congratulations, you got lucky; your comment is an example of survivorship bias. You happen to have the technical know-how and people to run your own infrastructure, but this isn't the default.

I'm not sure if it's a good thing if no software developer in a company has the skill to install and maintain an application on a linux server. To me that's no additional knowledge for a developer, but one of the basics they should know anyway.

Except it's more complex than that.

Ideally you want to have multiple runners, and for them to scale so that developers aren't endlessly waiting in job queues. On top of that you might want proper network segregation, and for it to be on a separate virtual network to your workloads to reduce your blast radius. You might have problems with connecting your server to third party SaaS outside your network, etc etc etc

A runner is super simple to set up and you once write a script and just deploy it a few times.

Virtual network is also very easy to set up with most providers (even with dedicated servers), there's pretty much zero config.

So far we didn't have any issues connecting to outside SaaS, but we host most stuff ourselves anyway, so that's not a common use case.

I run Gitlab runners for my social group.. It took about 20 minutes to setup and I don't think there has ever been a maintenance period for it other than a nightly upgrade/restart that is automated. I use docker for network isolation.

There is 'running a basic linux server' then there is running a secure, scalable and reliable server for the whole org to use. The latter is the job of a sys admin, not an developer. Sure I could probably figure it out, but I am no expert and it would be a waste of my time.

It worries me that running a monolithic web application + DB on a Linux server is now seen as arcane, specialized knowledge. If you think this is complex, I'd be worried about how you do the actually complex stuff like data security, etc.

Engineers have to understand the business, talk with other stakeholders, design systems, write code, write tests, support the production environment, etc, etc, etc.

Why would decisionmakers want to add outsource-able ops overhead to that very full plate?

It probably depends what sort of software your company builds. A frontend web design shop might well have developers who have little or no relevant experience for self-hosting something like Gitlab.

If you have the technical knowhow to develop complex applications and do application and systems level administration for your own application then you can run a gitlab on prem installation. You already have most or all of the required knowledge.

This.

That's how you do it (and also how many open-source organizations have done it) as I have said and predicted years ago here going against 'centralizing everything to GitHub' [0]

Fast forward precisely 2 years ago since that prediction was made in (April 14th, 2020), GitHub has become (unsurprisingly) unreliable, especially for those that went 'all in'. Self-hosting for your own organization seems to have been more reliable in the long run.

> Seeing constant issues with Gitlab and Github products just reinforces that decision.

This also. Especially the SaSS ones.

[0] https://news.ycombinator.com/item?id=22867803

At my start-up we are all in on SaaS because we don't have enough engineers to self-host our things.

The previous F500 company I worked at had enough engineers to self-host their things but thought SaaS was also the future and migrated a lot of their key tools and services onto these cloud platforms.

I wonder if both my start-up and the F500 company will suffer in the long run because of these decisions...

Gitlab in particular is unbelievably easy to self host. At my last gig I ran it on a t4g Debian instance with unattended upgrades and then just rebooted once a week at like 3am, literally never touched it again and it costs nothing to run.

Don't enable unattended updates for GitLab. The migrations don't always work and sometimes need manual work. (This is rarely the case, but can happen.)

(Also, when upgrading GitLab, make sure to upgrade version-by-version and always wait for background migrations to finish before upgrading to the next version. You can check background migrations at /admin/background_migrations.)

Interesting, thanks for the heads up - We didn't have any issues while I was there and probably would just restore if we needed to, but nice to know the footguns.

interesting, this isn't my experience. I've been running a gitlab instance for about 300 people for about 7 years now. :\

To be fair though I'm using docker and watchtowerr to keep it updated, there was 1 time where I had to use an older docker image because the latest wasn't working.

For those like me who went familiar with it: https://containrrr.dev/watchtower/

This happened to me exactly _once_ in the last three years of successful monthly update on our self-hosted GitLab instance. I got an error because the backup script expected a folder where no folder was. Updating GitLab is a breeze!

It’s incredibly dumb that it allows you to start the next upgrade before the background migrations finish.

Not unrecoverable though, so saying you need to worry about it may be a bit too alarmist.

But how many people would know what to do if something did happen?

Worst case you just spin up a new instance, install and restore the backup. Won't take more than 1-2 hours (less if you have part of it automated).

There was a fair bit of ops expertise on the team, so I don't think it'd be an issue. If there was none, I'd probably think twice.

If you have enough engineers to develop a product, you have enough to host your own Gitlab. Most engineers are expected to understand some ops anyway for CI/CD, setting up Gitlab really isn't harder than that (from experience). Especially with low usage, these instances basically never go down.

Most of the overhead is managing users, permissions etc, that won't go away with SaaS.

Yup, after the first series of outages (which were much worse for us in Asia than for companies in the US, because it seemed that nobody at GitHub noticed everything was down until the US woke up) we moved all our code hosting from GitHub to self-hosted Gitea (our CI/CD was already self-hosted with Tekton), and set up mirror repos from GitHub for all our dependencies too.

Before we were losing productivity for an entire day due to GitHub outages, now we only notice because there's a post on HN :)

As for the cost of self-hosting, we already host our own services, Gitea is no more complex than any of those. I wouldn't recommend self-hosting to a company with no backend/ops capability in-house, but for any other company it's fairly obviously a better choice IMO.

I thought gitlab onprem was an enterprise feature? You’re saying it’s cheaper than cloud?

Gitlab has enterprise and community versions that are self hostable. You may be thinking of GitHub.

Gitlab is great, the downside is managing updates and frequent vulnerabilities found

I find it concerning that the incident is marked as resolved and the incident history suggests that have corrected the billing.

We still have a $7m dollar balance on our account, so the issue clearly is not resolved.

Providing incorrect information on status pages makes it hard to trust in the future.

Seven... Million? Though it says the incorrect billing is still not fixed there.

I understand it differently. They said 7 hours ago it hadn't been fixed but afterwards they said the issue is resolved. I would understand that this means all issues have been resolved. Otherwise I don't get how they can close the incident.

9 hours ago they posted "We have started a cleanup of incorrect billing information and expect that to be completed in a few hours."

2 hours later they marked the incident as resolved, but they clearly have not completed the "cleanup" yet.

status page :

Resolved This incident has been resolved. Posted 7 hours ago. Apr 14, 2022 - 01:28 UTC

Update Actions and Codespaces are operating normally except for incorrect billing reports. We have started a cleanup of incorrect billing information and expect that to be completed in a few hours. Posted 9 hours ago. Apr 13, 2022 - 23:36 UTC

Update We've applied a mitigation to unblock running Actions jobs and using Codespaces. We are currently working on addressing the inconsistencies in billing information. Posted 11 hours ago. Apr 13, 2022 - 21:07 UTC

To provide a bit more info, I woke up to a $32m email, but it had already been fixed when I checked enterprise dash a few hours ago.

I’n fairly certain at some point today we owed 6 billion :P

ours was $3.1 million, luckily our account shows the correct billing.

What is the typical relative balance on your account? Does $7m mean 7 million USD? Over what period of time?

What exactly do you work on if you’re paying 7 million to just GitHub?

You have parsed the comment incorrectly. The OP is saying that the $7m is an error that is yet to be resolved.

To put these billing errors in context, our account suddenly showed ~$12,000,000 in usage fees. I saw some others posting bills high as 100s of millions. Pretty heart attack inducing.

That these were so scary shows the downside of usage-based fees. If you run on dedicated hardware, you always know what you'll have to pay, no matter how you utilize the system.

The risk of surprise bills, even with the advantage of higher scalability, is one of the major factors to put me off many cloud services. I wouldn't want to run a business where you always have the risk of some bug or unforeseen behavior destroying your budget.

I've said this over and over again. There should be laws in place enabling a person or company to set a hard limit on spend.

It would be a pro-innovation law because people would then have the confidence to innovate.

I also think it'd be good for companies. If you usually pay (say, numbers don't matter, except that AWS or whoever doesn't care until they get bigger I suppose) 1k-3k and set a 'don't surprise me too much' limit of 5k, then when something unexpected happens, instead of having a bad time pleading with them to let you off 12k or whatever and feeling cautious about using it again perhaps with some sour taste even if they do let you off, you think 'woah, glad I have the limit', and pretty much get on with your day; AWS or whoever gets an unexpected 2-4k instead of writing off 9-11k, and keeps the happy customer (at the customer's expense instead of its own).

I agree, unless... It could be that they're making most of their money from spikes. It would be interesting if that were the case.

I don't think so, surely most comes from far bigger fish than the type to have (not have time/resources to avoid) spikes and care about them.

I agree cloud providers shouldn't be able to bankrupt their users due to high usage. IMO it should be mandatory to set a limit it makes sense from a finance side.

In my case my cards don't have $12 million of a limit so they could not actually charge me that amount of money. It makes sense if you use invoice type billing i.e. like a company but as an individual these insane bills can't actually run.

they wouldn't be able to charge your CC but this isn't really relevant to what you owe them

they can still attempt to come after you for that $12 million

Of course but I cannot ever pay that amount so what would be the point from their side. It would be more sensible to stop after they run out of money on my CC.

These emails we got were about billing limit (which is meant to be a hard limit on github) being hit too. Either the bug ignored such limits (ours was in the low thousands) or it created charges so fast that the limit couldn't apply anymore.

You didn't read the blog post from GitHub saying they were switching to Zimbabwean dollars because the dollar is falling out of favor as the reserve currency?

Damn.

Well, when you owe $10,000 it's your problem, when you owe $10,000,000 it's their problem :D

I think this code wasn’t supposed to go into effect until Microsoft moves beyond their friendly stage and moves into maximum revenue extract stage.

yeah, better than coffee ;) ... we had $78,610,108

Our account is still showing $188,536,934.07 overage.

The amount was so ridiculous that it didn't replace coffee for me. :-)

Anyway, it's already fixed for our account.

Oh billing :( My own disappointing incident was metering customer usage in Bytes when billing was setup for Gigabytes. A couple of “concerned” customers noticed that super quick when the new functionality was turned on. Having to sort that out at 3am highlighted how many “business process” dependencies dont have the same level of operational response as our systems. On the plus side it highlighted some areas to improve continuous monitoring (and alerting) of what customers are metered and billed.

This type of thing scares the shit out of me everytime using cloud. The possibility of something idiotic happening and me losing my life savings is just too high.

That's why I use a cloud provider that doesn't charge for bandwidth usage and I use a fixed set of "stuff" such that I always know what my fee's should be.

I understand for big companies with millions of users they can't operate in that way but for me risk vs reward is too high.

> I understand for big companies with millions of users they can't operate in that way but for me risk vs reward is too high.

They absolutely can. If anything, big companies have more resources to build their own cloud based on on-prem or rented bare-metal than smaller companies. It's just all cargo-culting.

The "cloud" is not some kind of unobtanium hardware you absolutely can't get anywhere. With rare exceptions, it's standard hardware you can buy directly from any server & network equipment manufacturer.

That's an error on my part sorry I meant can vs can't.

Is it just me or are the incidents, outtages, security breaches just raining down on major services since a couple years?

No, it's just you. This has been happening since the cloud became a thing. What is changing is that the severity is going up because more and more parties are now dependent on these cloud things.

Wait until we have self driving cars that are somehow cloud dependent and in the middle of rush hour AWS or GCP decides to burp.

We spent three hours trying to track down what happened, thinking that an account had been compromised, before GitHub acknowledged this.

Our usage report said that the egregious minutes were run on deleted repositories so we were very concerned.

These outages and incidents are getting very tiring, and costly.

Less than a month with another incident with GitHub Actions. Last time that happened was two weeks ago. [0]

Not looking good for going 'all in' on GitHub products or 'centralizing everything' to GitHub as I predicted. [1]

I expect them to go down at least once every month guaranteed. Not 5 times in two weeks. I don't think they can handle a month without a serious incident at all.

[0] https://news.ycombinator.com/item?id=30841387

[1] https://news.ycombinator.com/item?id=22867803

This "incident" has no material effect beyond the number on the billing dashboard.

Actions were unavailable yesterday for folks, as the GitHub incorrectly thought that people were out of credits.

Actions fell over yesterday afternoon.

In our case first sign was Jenkins not kicking off builds on push.

Any idea if this tangentially impacted LFS bandwidth somehow? We're seeing a huge spike in LFS transfer starting yesterday afternoon. The spike doesn't seem to correspond to CI builds cloning the repo (because it was sustained over night after our devs had signed off). No notable changes to the repo, automated processes, etc.

We've had to purchase additional data packs to unblock CI builds and folks pulling from the remote and assume we will need to purchase more. 11 days left in our billing cycle...

Morning coffee, check Daily github incident, check

Super cool !