News for hackers. Stuff that matters.
Re:Incident with GitHub Actions and Codespaces by qeternity on Thursday April 14, 2022 @02:54AM (#31024079)
We switched from self-hosted Gitlab to Gitlab.com in the name of reducing infra burden...my oh my do I regret that choice. Same experience to you - no downtime that wasn't planned when we self-hosted. Load was very predictable, and ran on k8s anyway. Much, much faster than the hosted version.
Re:Incident with GitHub Actions and Codespaces by dx034 on Thursday April 14, 2022 @03:03AM (#31024122)
And I don't get what infra burden there is. We run Gitlab, Mattermost, Databases etc and running/maintaining those is still just a very minor part in the work of some developers (who like and are good at infra). Put everything behind Tailscale/Cloudflare Tunnel and your attack surface is very low to start with. Most deployment and patching can be automated anyway, so there's not much left. In the end we save a lot of cost, avoid lock-in and keep much more knowledge in-house.
Re:Incident with GitHub Actions and Codespaces by qeternity on Thursday April 14, 2022 @03:54AM (#31024371)
It’s just one more thing to backup, test, monitor, etc.

Upgrade migrations didn’t always succeed so then you need to go back and remember what you did last time to fix it.

It’s a cognitive burden and distraction. In theory it’s cheaper to pay Gitlab to handle for us. Except that now we lose more time due to Gitlab.com outages than we spent maintaining on prem.

Re:Incident with GitHub Actions and Codespaces by Cthulhu_ on Thursday April 14, 2022 @04:07AM (#31024427)
That's exactly it, it adds a list of nodes to the "required knowledge" graph of your company, adding job positions, expenses, people, and (operational?) risk to your company that I'm sure already has its own list of things to worry about.

There's so many companies out there whose infrastructure is run by this One Guy. Any company needs to assess whether they can risk losing their One Guy (see also: Bus Factor), or if they'll just have to swallow the cost and (some) risk of (some) downtime by using a service provider.

Re:Incident with GitHub Actions and Codespaces by Nextgrid on Thursday April 14, 2022 @05:46AM (#31024886)
But is it much different than the usual CI or infrastructure house of cards/YAML files?

Every company has their own way of doing cloud-based infra, CI build pipelines (in their favorite CI provider's dialect, since those aren't portable either), etc.

Every time I start at a new place I have to reverse-engineer what the previous guys did; there is no standard. By that benchmark, one extra thing to document/keep in mind isn't that big of a deal, especially considering GitLab (installed using their standard "omnibus" install on one of their supported distros) is likely to be more standard than all those custom Terraform/CI pipelines/etc.

Re:Incident with GitHub Actions and Codespaces by dx034 on Thursday April 14, 2022 @04:32AM (#31024542)
Any decent developer should be able to install and maintain on-prem Gitlab. It's not harder than running most managed services on AWS (I'd say it's simpler). So you already have the people, you just need a few minutes (literaly minutes, not hours) of their time per week/month.

There's operational risk, but that doesn't go away if you shift to Gitlab.com infra. Just in this case, you can control all of it.

Re:Incident with GitHub Actions and Codespaces by qeternity on Thursday April 14, 2022 @05:23AM (#31024770)
It's really not this simple. Even if it were just minutes (it isn't, but agree it's not huge overhead) you still have the operational risk when things inevitably go wrong. Now you're impacting your orgs ability to be productive (super expensive) and you incur the direct cost of fixing. Who do you think is going to be better at triaging Gitlab: someone looking after on-prem once a month, or Gitlab's team that do this full time every day. That's why people pay for SaaS.

The problem with Github and Gitlab is that it's actually turning out that those companies aren't that good at running their own product, which is tilting the economics of the above.

Re:Incident with GitHub Actions and Codespaces by Nextgrid on Thursday April 14, 2022 @05:42AM (#31024868)
> someone looking after on-prem once a month, or Gitlab's team that do this full time every day

Keep in mind that Gitlab.com sees much more load & activity than your on-prem equivalent, and their maintenance schedules don't line up with yours. Ideally, you don't want someone messing with your VCS/CI "every day".

Your on-prem is not only very unlikely to need any of the moving parts that a world-scalable system needs, but any maintenance can also be planned out-of-hours to make sure eventual operator errors don't cause downtime during business hours.

Re:Incident with GitHub Actions and Codespaces by Aicy on Thursday April 14, 2022 @04:52AM (#31024637)
In theory, yes, but in practice most of the difficulty in maintaining an on-prem Gitlab will be that individual company's bespoke installation of it.
Re:Incident with GitHub Actions and Codespaces by Aeolun on Thursday April 14, 2022 @05:34AM (#31024825)
> Any decent developer should be able to install and maintain on-prem Gitlab

By this measure, a lot of companies have zero decent developers.

Re:Incident with GitHub Actions and Codespaces by orwin on Thursday April 14, 2022 @06:49AM (#31025187)
I might be skewed because I do a lot of DevOps, but this is clearly one of the easiest thing to do for any half decent dev. You juste have to follow one half hour tutorial (and have a very basic knowledge of VLANs, DNS and routing, maybe? Depending on your internal network). The hard part is making sure your downtimes are low or inexistants, and if you this, hire a devops.
Re:Incident with GitHub Actions and Codespaces by dx034 on Thursday April 14, 2022 @04:26AM (#31024511)
Backups tend to take the most work, especially if you verify them periodically. But at least that means you have backups. I'm not sure I would want Gitlab to have all my data without a local copy for such important information.
Re:Incident with GitHub Actions and Codespaces by Nextgrid on Thursday April 14, 2022 @06:05AM (#31024966)
An issue everyone seems to overlook is that an SaaS typically needs to be scalable to the entire world while your on-prem service pretty much never does.

The SaaS will have more moving parts & operator activity which in turn makes outages more likely, and that "operator activity" can't be planned for unlike on-prem where you can schedule maintenance out-of-hours to make sure any accidents don't affect the business.