HN can be funny sometimes. GitHub got a lot of hate about a year ago just for not releasing new features. GitLab cost everyone a day because their backup/ops practices were silly, and everyone loves them more.

I've screwed up before, and I sympathize/empathize with their ops folks, but this should make us think about plan B in case something like this happens again.

> GitLab cost everyone a day

GitLab isn't popular because of the stability of its cloud platform. It's popular because you can install your own instance for free practically anywhere with minimal effort.

I run GitLab CE on a box in my server closet for projects that involve livelihoods.

I know (because I use it and also host it at home, and like it). The point of my comment was HN's reaction.

Remember that HN is not a single-minded mob, during each incident a particular group was vocal on a given subject but that group may not have been the same individuals who were vocal each time.

Fair point, thank you.

You can also look at RhodeCode CE - supports git/mercurial/svn and has a really nice installer you can use to keep things up to date.

>It's popular because you can install your own instance for free practically anywhere with minimal effort.

If that is the reason why you use Gitlab, then why not try gitea or gogs? gogs is written in Go and provides a docker image or a drop in binary

I have been running Gitlab CE for around three years now and I am very happy with it. The interface it good. The access management works very good. And the best part? Keeping it updated is very easy on a ubuntu box. Just a couple of very easy commands.

I am so happy with Gitlab that I didn't even install Gogs even once to give it a try, though I know about it since it's initial days. For me Gitlab CE just works. Unless Gitea shows a 10x feature I am very unlikely to shift away from Gitlab CE.

GitLab is a lot more feature packed than Gitea/Gogs. Gogs is lightweight, good for personal projects, but if you're looking for something to deploy company wide with integrated everything, Gitlab is the way to go.

I think this is changing at a pretty fast pace ... well for Gitea anyways. It also looks like Gitea is lighting a fire under Gogs, as they appear to be iterating at a faster pace as well. Here's a very quick breakdown of what's going on.

Activity for the last 160 days. There were 175 commits to gogs and 720 commits to gitea.

https://gitsense.com/gogs-gitea/commits-160days.png

Activity for the last 60 days. There were 109 commits to gogs and 262 to gitea.

https://gitsense.com/gogs-gitea/commits-60days.png

https://gitsense.com/gogs-gitea/changes-60days.png

https://gitsense.com/gogs-gitea/changes-files-60days.png

The options and vendors directory are unique to Gitea and they account for a lot of the changes within the last 60 days. I was told the vendors directory is used to store dependencies but I don't know what the options directory is used for. And as the following shows, they account for a lot of the files touched, in the last 60 days.

https://gitsense.com/gogs-gitea/changes-options-vendor-60day...

Based on what I've read on Hacker News, the developer behind Gogs, tends to merge in changes in spurts, so it's hard to tell if this recent flurry of activity is a spurt or not. In this 365 days of activity, you can see the 3 spurts for Gogs so far.

https://gitsense.com/gogs-gitea/commits-365days.png

Regardless of whether or not Gogs will continue to develop at an increased rated, it looks like Gitea will.

Gogs is pretty sweet. After hearing about it here I installed it on my old home server with constrained memory because I couldn't meet Gitlab's (IMO) obscene memory requirements. It's perfect for my needs as a hobbyist and I even use it as a secondary Git server from work. It's also way, way faster than a self-hosted Gitlab or any public Git server, so even though my current server has more than enough memory to run Gitlab, I still use Gogs.

The only (minor) issue I've had with it was when I tried to push the OpenCV repository to it, just for the hell of it, on a heavily constrained VM (Debian with 256 MiB of RAM). The poor thing just couldn't handle it without crashing until I upped the memory to a couple gigabytes.

It might be related to git and not Gogs/Gitea itself.

I've found Gitea to handle larger repos fine after running the following:

  git config --global core.packedGitWindowSize 16m
  git config --global core.packedGitLimit 64m
  git config --global pack.windowMemory 64m
  git config --global pack.packSizeLimit 64m
  git config --global pack.thread 1
  git config --global pack.deltaCacheSize 1m

This reduces the memory used by git during certain operations.

The one who erred (and recovered) once is more valuable than the one who never made mistakes.

There is no basis for that assertion.

This is basic risk management. What matters is error survival, not absence of errors. Moreover, total absence of errors means accumulating dangerous risk. See NN Taleb, for example.

Sounds defeatist to me. Especially when we are not talking about literal moonshots, but backup and restore procedures. The IT equivalent of seatbelts.

I think at this point, recognising that software and software deployments invariably have bugs/issues is just realistic. The only setups that are free of them, are setups where the problems havent been discovered yet.

A well known library like curl, has had CVEs.

A not known library that also addresses the same circumstances as curl might have none listed.

Which would you prefer to use?

That's a cross purpose argument

The one with the CVEs because it has had eyes on the source enough to generate them. The devil you know and all that.

Exactly. They've made mistakes, and can recover from them.

The other hasn't had that chance.

Thus, try the first, as it is better tested in the world.

That's very different, though. I trust the company/organization that had to fire an employee for a horrible mistake. They are going to be very careful with who they hire or let near the code in the future, and had much more to lose than a single person.

But I don't trust the employee that was shown forgiveness for a horrible mistake. Some might learn to not repeat what they had done, while others learn that they can get away with things through the magical power of phrasing the situation in a positive light. And some might be mistake-makes-for-life. Not all people come out the other side stronger.

I see this play out with regularity in the board game hobby.

Manufacturer A: perfect product. Manufacturer B: omits some pieces, expedites replacements.

People will love and extol the amazing service virtues of B, not A.

There's another reasonable takeaway: GitLab will probably now prepare for this sort of thing better than the average competitor who hasn't been bitten by this.

Borges: "In Alexandria, it has been said that the only persons incapable of a sin are those who have already committed it and repented."

We'll certainly learn from this, post-mortem write-up issue is in https://gitlab.com/gitlab-com/www-gitlab-com/issues/1108

Every corporation ever has screwed up like GitLab if not more.

They are just better at hiding it, smoothing it over and lying.

That's Internet culture for you: being lovable trumps pretty much anything else.

Nah, no need to think about backup now.

Stuff like this only happens once :) haha,

To be fair, I suspect gitlab will a adopt a 'never-again' policy towards making sure their backups work.

>GitLab cost everyone a day

That's just nonsense. In most cases, the local repo should be more than enough to continue work.

Only for development right? If you were someone that handled issues for the team, or depended on webhooks triggering builds, you were still blocked.

I'm not trying to skewer Gitlab, I host an instance at home. I've also nuked 30% of the ports on our openstack cluster and screwed up everyone's day. I admire the transparency. I just wanted to call out HN's reaction, vs way smaller (technical) issues involving GitHub. But someone did point out downthread that HN's not a hivemind, so there is that.

The actual problem was with their DB, not the repo storage so I'm not sure what the actual fallout was, but people lost things like issues/merge requests, so I can see how workflow would be interrupted.

Yeah, kind of defeats the whole D part of DVCS when you centrally-host everything. GitHub is great for creating an open-source watering hole and becoming a de-facto standard for open source project hosting, but bad for mission-critical stuff if you're relying on something like that to be 100% available, especially when using a technology which is designed to be distributed.

It would have been really cool if GitHub had fully bought into the distributed model and built the wiki/issue tracker/etc such that they were included in the repository, so that they were also decentralized.

The "Plan B" is to have a local Gitlab instance and use the mirror feature to mirror it to Gitlab.com and/or vice versa.

You should always have 2+ production nodes in case one goes down.

Just to stress one thing, while mentioned in their report, most people don't comprehend:

Gitlab was _one_ (last!) final step from complete data loss of everything. One. At that night, there was quite a long moment there had only one copy (and 6 hours old). Every other backup was missing/notworking/deleted.

This is scary.

This is the fact that deserves more attention. The operator did a manual backup before work which was the only available one.

I find this to be an eye opener. The real problem was dodged by doing the right thing in the proverbial last minute (6 hours).

In other words, there is an alternate universe in which Gitlab is now categorically dead, all because that employee didn't make that single backup.

Didn't they had two copies? (staging server and LVM snapshot)

I am sick of the "They are transparent so it's ok" argument. The fact is this type of thing should not happen anymore.. I get it, stuff fails but it's not acceptable to lose production data at this scale and save face by being transparent. That was ok back in 2003.. Not anymore.

Compare with Github and their 98.935% uptime over the past month: https://status.github.com/graphs/past_month

Neither should be exclusively relied upon for business-critical services. Ask Github about their production backups and DR plans some time.

I still remember the outage where github ran their tests against production - which started by TRUNACTEing everything before loading fixtures.

I think "everybody learns the hard way" is just one of those things with operations.

Probably they cut cost by hiring a group of really incompetent set of developers because they are a "startup" and "transparent" and "its ok". Nothing could make state of the company more "transparent" than that. xD

I don't think this affects to many of the larger shops, since they tend to host their own GitLab instances (typically for security, compliance, legal, etc.) My current shop runs GitLab community edition on an internal cluster.

Transparency? Gitlab is the company that interviews people and rejects them on basis of salary, they've been doing this for awhile, every few months they call a group of people, waste their time and then deny based on the salary. In my case, I've confronted them that USD 100-120k is the market average but they had this stupid startup argument that doesn't make any sense to me.

> USD 100-120k is the market average

In the US maybe, but given that they're a remote company with people all over the world, they don't necessarily have to live by US standard (especially when it comes to spending!).

40k USD is a VERY good developer salary in Argentina, and I'd bet that in other countries a lower figure might make the cut too. I can definitely understand why they'd hesitate to pay thrice that amount.

I believe it's over 5 times the average salary.

Yes, but if you look at the average here its not really outrageous or "5 times the average salary" as you said, here: https://www.indeed.com/salaries/Gitlab-Salaries

And its not just me, they've done this in the past as well: https://news.ycombinator.com/item?id=10924957

Also when you work for a remote company, they are cutting cost in terms of office and all which should reflect back in the salaries, the whole idea of working remotely was to mutually benefit both the employeer and employee and not just gitlab using their whole startup argument to cop out when they want on that "truly remote" so-called transparent company.

Like I said, they're a remote company, and therefore can hire people in cheaper regions, and don't have to adapt to more expensive markets (like your own).

I fail to see how this disproves it, it merely proves that you're not a good choice for them (because you live in a, relatively, expensive region).

> because you live in a, relatively, expensive region

If what OP is saying is true, then the company might as well be upfront about it.

> If what OP is saying is true, then the company might as well be upfront about it.

Regardless of that, the average salaries are posted online and they seem to suggest quite the opposite to what they offer people outside of US. It's just an indication of how much of a bully culture they have in negotiations or discriminatory, they might as well just outsource the site and not have a team of their own at all.

Most of their staff is from US/UK, but they just do not want to pay fair wages, which is why I posted the other thread where it was highlighted more.

Is it more fair to pay you $120k or more fair to hire three people in a lower cost area?

This argument sounds more selfish than fair to me.

It's unfair to pay low to some and high to the other (as I pointed out earlier most of their employees are in US who are getting paid the average US salary).

A company should follow some rules to neutralise the discrimination between employees, it shouldn't be all pick and choose and take advantage where they can, that's not really the reason one should have a remote distributed team to get cheap labor.

What's wrong with that? Isn't it obvious that salary is an important factor in hiring people?

Transparency is not a substitute for competency. Both are important, but I'll take the latter every time.

Yeah it made me not consider using them for anything other than toy projects.

Why?

Everyone has their own copy of the gitlab remote.

Not everyone has their own copy of the project's issues/pull requests/settings/webhooks/CI config/etc., which the incident affected (and 6 hours worth were lost). Git repositories themselves do have the happy side effect that at least someone on your team has a full latest clone on their machine, but that's only because the source code is what you're there to use every day. Even if GitLab had a great "checkout your issues/etc. as a git repo" story, not many users would have an up-to-date copy backed up 100% of the time.

Right, there was data loss, and it was not trivial.

Does that mean you should only use gitlab for toy projects? I don't think so.

I think they'll quickly learn from this.

Maybe a dumb question but couldn't they be included in a hidden folder in the root of your repo or something? That way they'd get replicated automatically every time someone does a pull.

It'd probably have to be on a separate branch, but yeah they technically _could_ do that. Maybe something like this? https://github.com/duplys/git-issues

We use gitlab.com for code reviews and CI, and we basically lost a day of integration productivity because of it.

Yes, our process is too tightly tied to a single service, but it happens because we don't have the resources to self-host our own solution. We love GitLab, but this has absolutely got us looking elsewhere.

> We love GitLab, but this has absolutely got us looking elsewhere.

If you love GitLab and don't want to self host you can pay them for GitLab Hosted (paid customers had no troubles today).

How many days of productivity are you going to lose migrating off GitLab ?

And how do you ensure you're not going to have the same problem at the next place.

And what will you do if the next place people doesn't tell you the root cause of a probable similar event?

The gitlab service is obviously important to your organisation, but not important enough to pay for their service, and incidentally, support a more robust environment?

Do you self-host any services currently & have they incurred any similar unexpected downtime in the past?

If you self-hosted code reviews/CI, would you expect having a similar downtime causing problem in the future?

I expect the answer to be "Yes" for most companies.

> I believe among other developers

I don't know. I'm very hesitant trying out GitLab now whereas I was interested before.

Getting better at ops? Checking backups regularly to make sure they work is not a new thing.

All press is good press.

For me it underscores one simple rule: you backups are useless if you don't regularly practice restoring from them.

Or I don't know, at least checking they exist

Any publicity is good publicity, right? :D

I think the "business people" are people who hide their technological inadequacy behind supposed business expertise. A businesses success is not determined by whether or not they use tools that adhere to business micro-cultural "values".

If "too much transparency" is a turn off for you, you're probably just an authoritarian trying to scheme and scam your way into profit, and you probably lack the confidence required to put whatever skill you think you have on display.

At my company we always heir on the side of transparency and liberally post to our status. subdomain whenever an issue is identified.

Recently, our Redis cluster failed and both master & slave host machines rebooted.

This caused a latency spike in our app, from roughly a 70ms to 400ms response time for less than 10 minutes.

We posted to status within 60 seconds and posted 3 updates within those 10 minutes.

The next day, a new customer (who hadn't gone live with the app yet) cancelled their subscription because the app was "not reliable".

I guess my point is that there is a balance to strike. Our customers are not tech-savvy in any way and treat any small issue as the end of the world. Maybe there's no need to freak people out for a minutes-long latency spike.

FYI

https://en.oxforddictionaries.com/definition/err_on_the_side...

Just in case you ever use that phrase in critical correspondence. Better to err on the side of caution.

Ah, thanks! It didn't feel right but was too lazy to check. Will definitely remember this for next time.

This demonstrates a remarkable lack of empathy. Try understanding why people on the business side do what they do—it's almost never out of malice or desired authoritarianism.

Working with people is hard, specifically because you have to get outside of your head and think about how they see the world.

I suspect the comment you're replying to meant that the business people might feel negatively about trusting the company that lost data, regardless of transparency.

It really wouldn't have been that hard to write your opinion in a way that doesn't disparage the character of the person (that you don't really know) that you're responding to.

There's no reason to hide behind a new account other than to be an asshole.