News for hackers. Stuff that matters.
Re:Data Loss at GitLab by activatedgeek on Wednesday February 01, 2017 @04:04PM (#13545156)
While this is disastrous, I still think Gitlab is the best thing happened to OSS. This could be taken as a rhetoric, but on a more actionable side, we must all learn from Gitlab's experience. Almost everybody experiences this issue, but very few come out clean.
Re:Data Loss at GitLab by nicpottier on Wednesday February 01, 2017 @05:57PM (#13546244)
I think you are confused.

The best thing to happen to OSS is GitHub, not GitLab. GitLab is just a fast follower and likely wouldn't even exist without the former.

I for one am happy to throw money GitHub's way for their role in so dramatically changing how we code.

Re:Data Loss at GitLab by iatanasov on Wednesday February 01, 2017 @11:43PM (#13548169)
I think you might be confused. One of the nice things to happen to OSS was the availability of kernel ( Linux kernel that is ) and git ( ... ) and both set very open models which allowed the OSS to trive. Github allowed for using git without knowing it .
Re:Data Loss at GitLab by JohannesH on Thursday February 02, 2017 @02:56AM (#13548895)
I think you might be confused. One of the nice things to happen to OSS was the availability of transistors and electricity. Both made it possible to make millions of logic calculations per second without knowing it. :)
Re:Data Loss at GitLab by mightbeconfused on Thursday February 02, 2017 @04:30AM (#13549246)
I think you might be confused. One of the nice things to happen to OSS was the availability of human beings to actually develop the concept of OSS and any underlying technology.
Re:Data Loss at GitLab by imiric on Thursday February 02, 2017 @11:44AM (#13551913)
Please leave these threads on Reddit.
Re:Data Loss at GitLab by dkural on Wednesday February 01, 2017 @08:28PM (#13547242)
Github itself is not open source. The irony.
Re:Data Loss at GitLab by grzm on Wednesday February 01, 2017 @08:36PM (#13547272)
There are plenty of useful closed source services and tools that can be used to support open source projects, or just software development in general. Indeed, they facilitate open source projects with an easy to use service. Is there anything they're doing that prevents or blocks open source development? They do have open source projects that they've made available.
Re:Data Loss at GitLab by dkural on Wednesday February 01, 2017 @09:37PM (#13547603)
My pizza joint does not prevent or block open source development either. Apple has open source projects that they've made available. Many people use Visual Studio and Azure for open source projects, or software development in general. I find it very easy to use. I hope Github makes a lot of money from enabling open source. Who knows, maybe Microsoft will buy it.

More seriously: Github is a business focused on turning open source community ethos into hard $$$. There is nothing wrong with that, and their model depends on people buying into the carefully constructed marketing of Github -hearts- open source. It's inconvenient for Github for people to point out that it is a closed source for-profit business with shareholders to please.

Re:Data Loss at GitLab by JonathonW on Wednesday February 01, 2017 @11:36PM (#13548147)
Meanwhile, GitLab is also a business focused on turning open source community ethos into hard $$$-- part of their core product may be open source, but they're not a non-profit.

(One might even say that Gitlab is even more focused on turning open-source community ethos into hard $$$, because GitLab takes open-sourced community contributions to GitLab CE and rolls them directly into their paid, closed-source GitLab EE. Not that I'm suggesting that's necessarily a bad thing; I just want to make it clear that this isn't a Mozilla-style "take contributions from the community for the community's sake"-type thing.)

Re:Data Loss at GitLab by yorwba on Thursday February 02, 2017 @01:35AM (#13548596)
> GitLab takes open-sourced community contributions to GitLab CE and rolls them directly into their paid, closed-source GitLab EE

Just FYI, the GitLab EE source code is available at https://gitlab.com/gitlab-org/gitlab-ee/. You're right about the "paid" part, though.

Re:Data Loss at GitLab by grzm on Wednesday February 01, 2017 @11:27PM (#13548109)
Many companies take advantage of open source software. Some support open source through support contracts. Some which develop software release some of it, and employ devs who also contribute to projects. I'm sure there are many other ways companies use and support OSS directly and indirectly; some, don't support OSS at all.

SaaS companies are in a position where it can make it difficult for them to release their core software because it severely undercuts their viability if it's easy for people to set up their own installations.

What follows is naïve speculation as to counter arguments:

One could argue that they could differentiate themselves on better quality of service and support. It's possible, of course, but there may be enough people willing to put up with lesser quality that it may not be feasible for them to survive.

One could counter that taking advantage of community software contributions can lower their own internal development costs. Maybe, but there are very real costs when managing an open source project, including not being able to move as fast as they may want to. For example, if GitHub wanted to make a change for expediency' sake that wouldn't be appropriate for the community code, they're saddled with maintaining their own fork, which adds considerable overhead.

Of course GitHub loves open source. Their business is to provide a service that benefits open source (and closed source, for that matter) projects by providing infrastructure to manage their code. Some of their current and former employees are well-known in the open source community. I refuse to see anything nefarious or untoward (or even the slightest bit bad) about this until I see them doing something that impedes or actively works against open source. I see nothing "inconvenient" for GitHub about this. Some people are far too quick to go out of their way to look for hypocrisy.

(And as for "turning open source community ethos into hard $$$": my goodness. Yes, they make money. I hope plenty to continue to support the business, improve the service, and pay their employees well. How many community projects pay nothing for hosting on GitHub? How many devs pay to host their own projects, or fork others? The only reason I've considered paying was to host private repos: not open source. It'd be interesting to see their revenue numbers, but I suspect most of their revenue is for enterprise installs for software that isn't open source. Any corrections as to this most welcome.)

And there are alternatives for those who want open source. GitLab appears to be a great one.

Re:Data Loss at GitLab by romanovcode on Thursday February 02, 2017 @04:58AM (#13549379)
Yeah, even their enterprise edition code is obfuscated like crazy.

For a company that praises open-source so much they really do try to be as closed as possible themselves.

Re:Data Loss at GitLab by dsacco on Thursday February 02, 2017 @11:51AM (#13551981)
Have you tried to reverse engineer GitHub EE?
Re:Data Loss at GitLab by drvdevd on Wednesday February 01, 2017 @06:26PM (#13546494)
I pay for GitHub and can't deny their positive role in OSS, but I also use GitLab and will happily continue (and love that they have an OSS edition). I don't think it's a binary choice. Personally I will often adopt multiple pieces of software with redundant features because nuance often dictates the right tool for the job, and I enjoy being prepared for that.
Re:Data Loss at GitLab by nicpottier on Wednesday February 01, 2017 @07:37PM (#13546954)
Absolutely, but the parent was saying GitLab changed OSS, and though that may be true to some extent, it is a pittance compared to the imapct GitHub has made.
Re:Data Loss at GitLab by manquer on Thursday February 02, 2017 @02:04AM (#13548692)
git itself is the real force behind the OSS impact. Github model while successful was merely replacing sourceforge and google code. People always had ways to host and share open source projects.

Github enables centralization of code which is not really inline with philosophy of git being fully distributed . Gitlab with its OSS model and the ability to host your own git service makes is bit closer to that vision than Github is.

Re:Data Loss at GitLab by Thaxll on Wednesday February 01, 2017 @06:43PM (#13546599)
You code differently using Git vs SVN? That's an interesting concept. It doesn't change the way you code it just changes the way we share code.
Re:Data Loss at GitLab by jasonwatkinspdx on Wednesday February 01, 2017 @09:14PM (#13547497)
Git makes multi-tasking on multiple branches far easier than SVN. In the SVN days I always had multiple checkouts and a pile of scripts for changing/updating to branches for handling a few regular tasks in that context. Once git won I was able to delete them and have a far simpler workflow.
Re:Data Loss at GitLab by gowld on Wednesday February 01, 2017 @06:56PM (#13546694)
It doesn't change how I code. It changes how we code.
Re:Data Loss at GitLab by nstj on Wednesday February 01, 2017 @09:23PM (#13547550)
Git absolutely changes the way I code. Cheap feature branching and easy rapid collaboration are huge in how they influence my coding practice and output. I wouldn't enjoy coding anywhere near as much if it weren't for Git.
Re:Data Loss at GitLab by VintageCool on Thursday February 02, 2017 @12:55AM (#13548439)
A few months ago I was developing for a breaking change and realized that I wasn't committing anything. That's very much an SVN mindset where merging is so painful that you culturally just avoid it altogether. Git's emphasis on being distributed and branching and merging constantly meant that, once I pulled my head out of my ass and started committing locally, I was able to enjoy the benefits of source control while developing my breaking changes without affecting anyone. And later, when I merged my branch into the master, it turned out to be completely painless.
Re:Data Loss at GitLab by flukus on Wednesday February 01, 2017 @06:48PM (#13546639)
Github was just a follower with a new SCM. The original would be source forge.
Re:Data Loss at GitLab by nicpottier on Wednesday February 01, 2017 @07:39PM (#13546969)
SourceForge pretty much failed where GitHub strived though. For all practical purposes they were just a hosted CVS/SVN and nothing more. GitHub really nailed the idea of community and that made all the difference. Git is incidental, though lightweight branching was probably an important contributor to their approach.
Re:Data Loss at GitLab by stevekemp on Thursday February 02, 2017 @06:52AM (#13549857)
While I can't disagree that sourceforge failed, it was definitely more than a mere code-host.

Sourceforge had integrated mailing-list support, bug trackers, page-hosting, and more.

There was a community there, it was just that each community was based around a particular project. There was little chance of a user of project A from interesting with project B. But I guess the same could be said of github.

Sourceforge failed in part because of feature-creep, and availability issues. But I think it would be unfair to say that it wasn't "social".

Re:Data Loss at GitLab by flukus on Wednesday February 01, 2017 @09:22PM (#13547536)
I know the execution was a lot better with github, but what did it really add that source forge, google code, etc didn't do? They all had bug trackers, mailing lists, etc.
Re:Data Loss at GitLab by will_hughes on Wednesday February 01, 2017 @10:19PM (#13547831)
The killer feature is stupid easy forking and sending pull requests with the one interface.

Found some tiny project that does almost everything you want, but has a tiny feature you can implement easy? No worries, fork, add it.

Even if it's never pushed upstream, it's easy for someone to find and perhaps use themselves.

I do this with things like zabbix templates/scripts and docker containers that are 99% what I want.

Previously there was no easy way to do this - I'd have to set up an entirely new sourceforge/google code/etc account, which is a lot more effort.

Re:Data Loss at GitLab by flukus on Wednesday February 01, 2017 @10:54PM (#13547961)
> The killer feature is stupid easy forking and sending pull requests with the one interface.

That's the part that wasn't really possible before git though. But you could always do an anonymous checkout and make your changes there,you could even do a pull request (via a patch). There was no need to signup for that, even though most of us had a source forge account (I still do) anyway.

Re:Data Loss at GitLab by kakwa_ on Wednesday February 01, 2017 @07:24PM (#13546870)
Well, even if their transparency in failure is admirable, I'm not completely surprised.

Just running an apt-get install gitlab gives me around ~350 dependencies. In those dependencies, I see python, ruby, nodejs, redis, postgres.

With a little Java, and a little Go, plus some admin scripts written in Perl, the picture would be mostly complete...

I may be a little harsh, but when I see a piece of software with so much complexity in it that it requires 3 stacks, I'm not completely convinced it's a well conceived piece of software.

Having 5 backup systems, with none of them working properly kind of falls in the same category.

Re:Data Loss at GitLab by Operyl on Wednesday February 01, 2017 @11:57PM (#13548226)
Err, isn't Git the reasoning for that perl dependency? Iirc, Git depends on Perl for --interactive..
Re:Data Loss at GitLab by rimantas on Thursday February 02, 2017 @01:49AM (#13548644)
There not many project these days that would not require similar stacks. At least one core language: be it Ruby, or Java or whatever. Then you will have a DB. Then you will have some fancy frontend which will sure depend on node js. Then you might have some KV store/caching so you get Redis.
Re:Data Loss at GitLab by tigershark on Thursday February 02, 2017 @02:07AM (#13548705)
I completely disagree. Normally you choose ONE core language and you use only that in a monolith application. I can admit different languages for the unit tests, like groovy on java side or F# on .net. Or using two languages on the same platform, like Scala and Java on the JVM and C# + F# on .net. Or one language server side with the web side written in javascript or in something else. If you are not breaking your application into micro services using several languages is just a very very bad smell. And in this case, apparently, they are using python, ruby, javascript, java, go and perl. I would say that from my point of view this is unacceptable and it is just a symptom of scarce planning, as we have seen in this incident where apparently nobody thought to plan for a backup test.
Re:Data Loss at GitLab by vorg on Thursday February 02, 2017 @05:21AM (#13549465)
Even using 4 languages is too many, e.g. Java and Scala for system building, Apache Groovy for build scripts and unit testing, and Javascript for web side as in your example. Go with something like Kotlin, which compiles to JS, Android, and JVM bytecode, can be used for Gradle build scripts, and has builder-style syntax useful for test scripts. A single language is better than 3 or 4.
Re:Data Loss at GitLab by dman on Wednesday February 01, 2017 @07:32PM (#13546918)
I would be surprised if most startups arent doing something similar behind the scenes.
Re:Data Loss at GitLab by dsacco on Thursday February 02, 2017 @11:59AM (#13552056)
In my experience of working closely with about 100 startups, most do not have that complex a tech stack. What I see nowadays is basically:

[Python|Ruby|JavaScript] backend + [JavaScript] frontend + [Postgres|Mongo] database + redis + [AWS|GCP] hosting.

Rarely, I see Java or Go instead for the backend. I cannot recall the last time I saw more than three languages in production for anything nontrivial. I've seen companies significantly larger subsisting on one language for backend (even in microservice architecture!) and one language for frontend. That's not to say there is no sophistication, just that the number of actual technologies in play is slimmer.

This isn't a comment on GitLab's utility or stability, of course. I haven't worked with them in this context, and I'm not a user. I'm just pointing out that, assuming those dependencies are all for GitLab and not e.g. git itself, that is quite a stack to maintain. I don't know if we can extrapolate that to a systemic issue with GitLab that caused a data loss incident, though. That seems uncharitable.

Re:Data Loss at GitLab by baq on Thursday February 02, 2017 @03:08AM (#13548937)
to be fair, nodejs is useful for npm even if you're only doing client-side js.
Re:Data Loss at GitLab by freehunter on Wednesday February 01, 2017 @09:10PM (#13547484)
That's what I thought when I read the report earlier today. A system with that much complexity is doomed to failure eventually.

I can understand "right tool for the job" but at some point it should all come together. An MVP can be hacked together from bits and bobs, but when it becomes a business it should be refactored to reduce complexity where possible.

Trying to maintain five seperate backup solutions, let alone trying to restore from all five of them, sounds like my worst nightmare. Trying to restore from one backup is often hard enough by itself.

Re:Data Loss at GitLab by iurisilvio on Wednesday February 01, 2017 @05:37PM (#13546100)
Exactly. today we're able to see a bad disaster recovery happening, but it happens all the time. I revised my backups and added some redundancy to it.

Everyone knows how much backups are important and talk about it all the time, but I bet a lot of companies don't do it right. It is expensive, don't add real value (except when it does), etc.

Today I added automated replication of backups from AWS to another cloud provider. Just in case...

I did this becauase recently, a local brazilian cloud provider (ServerLoft) didn't paid his server (Equinix) and went offline forever. 16k companies went offline without time to recover anything there.

Re:Data Loss at GitLab by Klathmon on Wednesday February 01, 2017 @05:47PM (#13546183)
As terrible as it is for GitLab, it's doing some good here as well.

I'm using this as a great example of a reason to bump up some of the fixes to some of our backups and replication issues up the priority list. And it's much easier to sell to some of the "higher ups" when you can point at a concrete example of how badly a misstep here can hurt.

I'm floored with their honesty and openness, I can honestly say I wouldn't be able to put this out there like they have... But i'm really glad they are doing it, and I'm really happy at the outpouring of support they are getting for it from people like 2ndQuadrant.

Re:Data Loss at GitLab by provost on Wednesday February 01, 2017 @04:33PM (#13545467)
As someone who hasn't had the opportunity to use Gitlab, could you expand on why you "think Gitlab is the best thing happened to OSS"?
Re:Data Loss at GitLab by mi100hael on Wednesday February 01, 2017 @04:43PM (#13545577)
It provides a lot of the same features as GitHub, only you can self-host for free, contribute to development, and all the other niceties of a free project. A lot of people (myself included, and OP I suspect) believe it's also a particularly good project because GitHub has too much market share and influence in the free software ecosystem for a for-profit company.
Re:Data Loss at GitLab by Vinnl on Wednesday February 01, 2017 @04:53PM (#13545681)
Note that GitLab is a for-profit company as well. It's just that it aligns better with the open source values.
Re:Data Loss at GitLab by djsumdog on Wednesday February 01, 2017 @04:59PM (#13545739)
..and they have a commercial offering too. Unlike some other things I've seen, at least their commercial offering isn't crippled and mostly has things that specifically cater to enterprise customers (like Active Directory integration. The community edition still has basic LDAP).
Re:Data Loss at GitLab by mschuster91 on Wednesday February 01, 2017 @06:32PM (#13546524)
> like Active Directory integration. The community edition still has basic LDAP

I manage our company's GitLab instance. It's connected to our massive AD and it works just fine - the only thing I really miss is LDAP group creation and assignment. GitLab uses only the name and email attributes from LDAP/AD either way, and I think if I have some spare time I'll just write a hourly cronjob that manages groups and assignments using the GitLab API.

Re:Data Loss at GitLab by Girlang on Wednesday February 01, 2017 @09:28PM (#13547576)
Plus github has an agenda that has nothing to do with code, and excludes large segments of the public.
Re:Data Loss at GitLab by i336_ on Friday February 03, 2017 @10:32PM (#13565502)
Your comment was killed (I just vouched for it) because of your unsubstantiated claims. Can you give some concrete examples of why you think this way?
Re:Data Loss at GitLab by tigershark on Thursday February 02, 2017 @01:52AM (#13548655)
seriously? almost everybody has FIVE different backups system and not one of them works? I don't know where you work, but in my job monitoring everything is essential. And a couple of times per year we do a disaster recovery test. Something like this is completely and totally unacceptable. It means that they never tested any backup.
Re:Data Loss at GitLab by FlorianRappl on Wednesday February 01, 2017 @04:28PM (#13545406)
Couldn't agree more. Also transparency is key here. Everyone makes mistakes, but being open about it does not only let customers (and employees!) know what's going on, but also gives other professionals the ability to learn from it.
Re:Data Loss at GitLab by neurotech1 on Wednesday February 01, 2017 @07:58PM (#13547094)
There are two types of web application providers. Those who have already had database mishaps causing data loss, and those that WILL. No matter how good, or our how large and 'professional' the company is, mistakes can happen.

Being open about it inspires confidence that they will improve in the future, instead of being quiet and having repeat incidents.