News for hackers. Stuff that matters.
Re:GitHub was down by rvz on Monday July 13, 2020 @03:09AM (#23817877)
14 Days ago, they went down [0]. And today it's happening again. Twice in less than a month.

Another reminder to self host via solutions like GitLab or Gitea. [1]

[0] https://news.ycombinator.com/item?id=23675864

[1] https://news.ycombinator.com/item?id=23676072

Re:GitHub was down by gregoriol on Monday July 13, 2020 @03:19AM (#23817963)
Could you provide details on how you plan to be more reliable with a self-hosted solution? what kind of archtecture would you use? how many people will be involved in maintaining?
Re:GitHub was down by koonsolo on Monday July 13, 2020 @05:25AM (#23818825)
This is an issue I see at a lot of companies make, even going further than self-hosting. For example replacing off the shelf software with an internal product.

"This is not good, we have to do this ourselves". And in the end it costs way more, and you end up with an inferiour solution because it's not your core business, and nobody really has any time to work on it.

Very sad really.

The first thing to ask is exactly what you asked here. I can't upvote this enough.

Re:GitHub was down by gregoriol on Monday July 13, 2020 @03:27AM (#23818016)
It's not that I'm against, the contrary, but in the last months I've experienced a physical server crash, a random kernel panic and a failed upgrade of GitLab: that all took 10/20 hours of work, didn't result in any data loss, but the downtime was clearly higher than what GitHub has (+ nobody on the team was on holidays at that moment), plus any maintenance like software or system upgrades has to be done outside team working hours to limit the impact.. real self-hosting is not that easy!
Re:GitHub was down by ashildr on Monday July 13, 2020 @11:32AM (#23821833)
A failed upgrade of GitLab? Can you give details - I got used to monthly Gitlab updates going very smoothly and now I’m a little afraid. What went wrong?
Re:GitHub was down by gregoriol on Monday July 13, 2020 @01:52PM (#23823512)
Last one we had was because the ssh connection was lost during the apt upgrade (gitlab omnibus ce).

Another one I remember was when some new configuration key was needed and prevented one of the components from starting (took ~2 hours to find what it was).

It doesn't happen often, don't worry that much though, gitlab omnibus works well, but is hard to investigate/fix because it's huge and not a tech stack we do understand well since we mostly use gitlab-ctl commands, so when something goes wrong it's hard

Re:GitHub was down by HelloNurse on Monday July 13, 2020 @04:23AM (#23818438)
No random high-risk infrastructure updates?

A server is more likely to go belly-up by accident than the presumably redundant GitHub infrastructure, but its life is much less interesting.

Re:GitHub was down by creshal on Monday July 13, 2020 @04:29AM (#23818481)
Git is decentralized by default, so all you really need is for your self-hosted version to not crash at the same time as github does. That's not too hard to pull off.
Re:GitHub was down by aabbcc1241 on Monday July 13, 2020 @04:17AM (#23818407)
If you just need the standard git functions, you just need an low-end linux machine with ssh server to "self host git"
Re:GitHub was down by dt3ft on Monday July 13, 2020 @03:20AM (#23817967)
I would choose self-hosting for small to medium size teams any day. I can't fathom why people choose not to self-host at this scale. Your data. Your control. Your network. Your infrastructure. Your responsibility. Are people becoming more afraid of responsibility these days?
Re:GitHub was down by dtech on Monday July 13, 2020 @03:26AM (#23818012)
Small to medium size teams should be hyper-focused on developing their product and bringing in customers and revenue. Spending time on commoditized infrastructure is only a good idea if that saves time in the short to medium term.

But hey, with the number of outages GitHub has recently that break even point is coming closer and closer.

Re:GitHub was down by douglasheriot on Monday July 13, 2020 @03:30AM (#23818048)
> Your control. Your network. Your infrastructure. Your responsibility

These are all the reasons why I don't want to self-host my own git. I can live with it being down every now and then. And when it is down, I don't want it to be my job to fix it. I've got more important stuff to worry about.

Re:GitHub was down by moooo99 on Monday July 13, 2020 @04:06AM (#23818334)
> Are people becoming more afraid of responsibility these days?

I wound't say people become more afraid, they just don't see the reason to bother. I'd choose GitHub or Gitlab for small teams at any time. I'd probably even be fine with the free versions.

I see little to no reason for self-hosting in a small team. I cannot imagine a performant server, bandwith and the employee it takes to maintain it to be cheaper than the 10€/month/user for a hosted solution.

Re:GitHub was down by Symbiote on Monday July 13, 2020 @04:19AM (#23818415)
From zero servers to one requires an employee with the skills, but from 100 to 101 (my situation) does not. It requires less than 1% more employee, since more is likely to be automated etc.

Many small companies already have VMs running other services.

Re:GitHub was down by creshal on Monday July 13, 2020 @04:42AM (#23818572)
> I cannot imagine a performant server, bandwith and the employee it takes to maintain it to be cheaper than the 10€/month/user for a hosted solution.

Gitlab and gitea don't have that high performance requirements, especially when you consider how absurdly cheap high-core-count CPUs, RAM and NVMe SSDs are these days… and that's assuming you don't just chuck it into your existing cloud infrastructure and call it a day.

But even self-hosted redundant gitlab setups for several hundred users can be done with cheap commodity servers (or dedicated hosting providers like Hetzner) for <5€/user/month, and maintenance is on the order of 1-2 hours per month.

Re:GitHub was down by rorykoehler on Monday July 13, 2020 @03:23AM (#23817995)
It won't help bring in revenue. Pointless to worry about that at the start
Re:GitHub was down by robjan on Monday July 13, 2020 @03:55AM (#23818254)
This assumes that develop / ops team is no cost and that your uptime would be higher (it wouldn't)
Re:GitHub was down by sergiotapia on Monday July 13, 2020 @03:31AM (#23818067)
it's literally a money sink for wishy-washy benefits if you're a small/medium sized team. It's why heroku became a billion dollar company, time+money is better spent elsewhere at the start.
Re:GitHub was down by jaekash on Monday July 13, 2020 @03:27AM (#23818030)
Self hosting quickly becomes a mess of weird gatekeeping and bad maintenance. Almost never have I seed well maintained self hosted infrastructure. Most of the time it is installed, gatekept and then left to bitrot without any upgrades or maintenance.
Re:GitHub was down by trulyrandom on Monday July 13, 2020 @03:49AM (#23818205)
Self hosting GitHub is an option as well. We've been self hosting GitHub Enterprise for years and it has had no downtime other than during scheduled updates (at night, when nobody's at work).
Re:GitHub was down by jaekash on Monday July 13, 2020 @03:54AM (#23818240)
> We've been self hosting GitHub Enterprise for years and it has had no downtime other than during scheduled updates (at night, when nobody's at work).

Is it free to self host like GitLab or Gitea? If not, what does it offer that you can't get from those?

Re:GitHub was down by joelhaasnoot on Monday July 13, 2020 @04:12AM (#23818374)
On-premise or own-cloud hosting is quite import for some customers.
Re:GitHub was down by reificator on Monday July 13, 2020 @04:59AM (#23818685)
> > Is it free to self host like GitLab or Gitea? If not, what does it offer that you can't get from those?

> On-premise or own-cloud hosting is quite import for some customers.

Yes... it is quite important- That's why people use things like Gitlab or Gitea....

But back to the original question, what does Github Enterprise offer that you can't get from Gitlab or Gitea?

Re:GitHub was down by jaekash on Monday July 13, 2020 @04:24AM (#23818445)
> On-premise or own-cloud hosting is quite import for some customers.

Yes it is, and GitLab and Gitea both offer that, which brings us back to the question I asked, what does GitHub Enterprise offer that you can't get from GitLab and Gitea?

Re:GitHub was down by justincormack on Monday July 13, 2020 @04:52AM (#23818628)
No it is not free, it is GitHub's main source of revenue.
Re:GitHub was down by dtech on Monday July 13, 2020 @03:12AM (#23817894)
self-hosting requires quite a bit of scale to be more reliable, otherwise you'll most likely still have possibly longer outages, just at different times.
Re:GitHub was down by Symbiote on Monday July 13, 2020 @03:28AM (#23818031)
That surely depends how much they use Github, and when they need the service available.

Our self-hosted build and artifact servers have not gone down without planning in many years. Planned outages are brief, for a small team it's not inconvenient to announce a 10 minute outage for an upgrade.

Re:GitHub was down by flukus on Monday July 13, 2020 @03:24AM (#23818000)
Very often that's ok if you can choose when the outages occur. With most you can because outages tend to be for regular maintenance and upgrades far more often than hardware failures. A small geographically concentrated company only needs 9 fives.

Even for bigger companies that need more uptime they can at least give a shout out to everyone that it will be unavailable from X to Y, it doesn't just go down randomly with no notice whenever github decides to push a release.

Re:GitHub was down by gregoriol on Monday July 13, 2020 @03:31AM (#23818061)
If you think self-hosting doesn't randomly go down, you have never done self-hosting.

In the last few months we had a server that physically crashed and had to be replaced, we had a bad kernel that paniced randomly and a gitlab upgrade that failed in the middle.

Is you team equiped and ready to handle these? If not, don't self-host anything critical to your company, or your costs, downtimes and losses will be much much higher.

Re:GitHub was down by dijit on Monday July 13, 2020 @03:44AM (#23818153)
this always comes up, and it's one of those things that's obviously true on the surface until you understand context.

I have a self-hosted gitlab instance and it has much higher uptime than github in the last year. At some point a weekly downtime is going to trump any potential random outages.

The major difference might be your ability to affect those outages (positive or negative) because humans are the primary cause of outages, and you can restore service to a gimped passable state if its your own thing.

Re:GitHub was down by vasachi on Monday July 13, 2020 @03:27AM (#23818024)
I mean, hardware nowadays is quite reliable. We have a simple server with three disks in raid1, and it basically just works. Gitlab has regular backups, so at worst we would have one day of lost work.
Re:GitHub was down by rplnt on Monday July 13, 2020 @03:17AM (#23817932)
You can self-host GitHub too. Or BitBucket.
Re:GitHub was down by jaekash on Monday July 13, 2020 @03:54AM (#23818251)
> Or BitBucket

But why would anybody ever do that?

Re:GitHub was down by rplnt on Monday July 13, 2020 @04:08AM (#23818344)
It's surprisingly easy to set up and run, even with redundancy, integrates well with Jira, Confluence. Was fast enough, stable enough, had everything one needed (offered more than a hosted solution such as github). IIRC kept git completely separate, so it worked even during updates or restarts.

Disclaimer: I passionately hate Atlassian products, but I don't feel like using Stash/Bitbucket is a bad choice in any way.

Re:GitHub was down by moooo99 on Monday July 13, 2020 @04:03AM (#23818317)
I actually enjoyed BitBucket for smaller projects. They offered free private-repos earlier than GitHub did, so that was the main sellingpoint for me.

The self hosted version of BitBucket comes in at a pretty reasonable price. I don't know that for sure (and I can't check thanks to GH being down) but I'd imagine the self-hosted versions for GitHub Enterprise are quite a bit more expensive than Bitbucket or Gitlab.