Then Dmitriy and Kamil (our CI lead) proposed integrating the CI coordinated into GitLab the main program instead of a standalone Ruby on Rails app. Again I was sceptical. But this has been a great success.
We have all the API's (webhooks, commit status API, etc.) to integrate well with Jenkins (and we'll make sure that we'll always play well with that, JIRA, Slack, and others). But having everything in one interface and one datastore makes for a much better user experience.
We've since doubled-down on this strategy. Bringing the entire software development lifecycle under one roof https://about.gitlab.com/direction/#scope During the webinar we'll show a demo video of how these components will soon work together. We think it is an amazing experience and everyone on our team is focussed to get it right.
Replacing Jenkins has been... a frustrating experience met with nothing but hollow shells of services. We want to self-host, we want something that can deploy our code, we want to be able to write arbitrary jobs not tied to specific github projects.
The Jenkins ecosystem is atrociously bad. Don't get me wrong, it's incredibly active, but it promotes a "just make a plugin for everything" attitude which results in things like, plugins to echo this or that, plugins to do a million things better done from the shell, etc. On top of that, the Jenkins UI is just... bad. It's ugly and it's a UX disaster. I could go on.
Yet, there's nothing better than Jenkins out there right now. Drone claims to want to replace Jenkins, but doesn't even have support for arbitrary jobs. I'd love more recommendations.
This is why, a while back here on HN, I asked if I could pick up the Gitlab job runner in golang and slap a different web UI on top of it that manages the jobs. This doesn't appear to be possible but I see the value in such a runner existing. Running the jobs is a critical component of such tools.
I just finished setting up a complete pipeline for a new business with my cofounder: Jenkins was a big pain point. Terraform was a pain point. Ansible was a pain point. Getting a vault solution set up and done right was a massive pain point. We've been striving to make things as simple as possible for developers so I wish you the absolute best of luck in what you're doing, I'm continuously impressed with Gitlab.
Email me if you have questions Sytse, I'd love to give more feedback -- email on my profile.
I'm not sure what you use case it but GitLab CI is pretty extensive and I hope it has everything you need. Please let me know if there is anything missing. If you want to host your code somewhere else you can still use remote mirroring to have GitLab do the deploy.
The GitLab Runner is a simple piece of software on purpose. This allows us to port it to many platforms. All the logic is in GitLab itself so it is easy to provide a lot of feedback about the status in the UI.
The idea of a runner that tests every commit on a repo is super useful. But Jenkins goes one step further by being essentially an arbitrary shell command runner. I use it for example as a CNC for running cleanup jobs, cronjobs etc.
For example, on Jenkins right now, we have a job which can trigger a bunch of load tests against our website. We can run this job indefinitely, give it arguments, it'll even run a local web server. By default, the job only runs the load test for ~10 mins.
The job is run periodically. When it fails, we get emails. After it completes, regardless of failure, it triggers a "cleanup" job which deletes the loadtest-related data from the website.
This is what I mean with "arbitrary jobs". I think it's absolutely fair to consider them out of scope, but if you want to replace Jenkins, this is what you're going up against.
Maybe you prefer the load testing code in the same repo as the application. In that case it should be two jobs that are triggered periodically I think.
I think what GitLab CI still needs is periodic jobs https://gitlab.com/gitlab-org/gitlab-ce/issues/2989 (right now the workaround is sending a build trigger for an external task).
I want to be able to just drop in some arbitrary command (e.g. df -h /some/build/agent/dir/ | grep 99%) that would not be worth it to me to write a script, commit it to git, get that deployed and running.
In fact, I AM a fan of this being in git, but the example I'd use is an improvement on TeamCity, where you can configure your job and project configurations to "commit to git" so that you can easily restore when you need to rebuild a server or reload a previous version.
Sooooo many use cases for arbitrary jobs. Many of which may be short lived, but want them automated in some way.
(speaking as a sysadmin/eng/devops/cloud dude)
The "code" in this case really lives in the Jenkins database, as it's a shell snippet. Admittedly I'm not a fan of this. However, since you're a FOSS company as well, you may have hit the case where you want to separate the code for your infrastructure from the code you're releasing in the general repository, but still build off master.
This is where Jenkins' model is quite good. I strongly dislike that it's so hard for Jenkins' metadata to be versioned in proper git repositories, but the separation of the actual code, and the CI-related code is great.
In my taste the shell snippet that is infrastructure specific should like in a small repository. An example of that for us is https://gitlab.com/gitlab-com/runbooks (although that is more process since we try to keep the code generic).
"any commits should be pushed to the upstream repository, and will end up in the GitLab repository automatically within an hour, or when a forced update is initiated."
"Wait up to an hour for things to happen" isn't a great story to pitch to team members :/ I'm also talking about at least twenty different repositories. I'll most likely have to continue using Jenkins for those projects but I look forward to using Gitlab CI on weekends.
(Not involved, just a happy user)
> I'd love more recommendations.We hope to make it easy to go to a single place for everything needed for development from issue boards, CI, analytics, etc. The unified UX for this tool is key and we are hiring a great team to support that goal.
Later today we're having a live event to talk more about that vision and answer questions. https://page.gitlab.com/20160913_unveilingmasterplan_landing...
Again, love the notes and how you're thinking about this. Happy we are aligned!
Edit: Sorry! Removed the signature. Habit :)
We have:
1. Extensive JIRA integration (and we're working on improving this) https://docs.gitlab.com/ee/project_services/jira.html
2. Jenkins support https://docs.gitlab.com/ee/integration/jenkins.html via our commit status API https://docs.gitlab.com/ee/api/commits.html
3. Slack integration https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/proj... and we're working on a Slack bot https://gitlab.com/gitlab-org/gitlab-ce/issues/20799
We realize that we should allow other tools to integrate well and want to open up as much as we can via API's.
We think it'll be a piece by piece migration versus all at once. In a perfect world, we have a tool for an all at once migration but perhaps that will come with time.
The CI solution in GitLab is inadequate for our needs; we require the flexibility and extensibility of Jenkins.
The review tool in GitLab has some significant gaps compared with standalone tools like ReviewBoard, and even compared with the one built into Bitbucket. It also has some baffling UX decisions, like the thumb/emoji buttons and the approve/merge buttons which appear in the exact same location as each other. Seriously, just scrap what you have and copy Bitbucket's review interface.
I mean, it basically just runs arbitrary docker containers. From my mostly amateur (when it comes to this, i'm a professional developer but I don't manage our CI system directly) point of view it seems like the gitlab CI is more flexible than jenkins not less.
The reporting is a big problem too— Jenkins has plugins which format and report on JUnit-style reports, including displaying images or other assets generated by tests.
Configuration. The config needing to be committed to the repo is an issue. For many use cases including ours, it's way better to have the CI configuration stored centrally somewhere, and deployed using Jenkins Job Builder (or the various other similar tools which exist for Jenkins).
Executor control. Jenkins lets you do things like put N slots on a buildslave and then have a job occupy M of them, as well as controlling what jobs can run where, and what the priority of different jobs is. This is critical for ensuring overall sanity and efficiency in the pipeline.
Jenkins isn't perfect, but it happens to be a better fit for our needs, and these four factors are all a big part of that.
We're considering parsing JUnit style tests https://gitlab.com/gitlab-org/gitlab-ce/issues/22098
We think storing the configuration in the repository makes it easier to collaborate on it (new branch is tested in the new way, master in the old way). Can you share your use case for this?
You can specify how many jobs you want GitLab Runner to run. It can even spin up more VMs automatically if you want that. You can bind jobs to runners with labels and specific runners https://docs.gitlab.com/ee/ci/runners/README.html
https://github.com/ros/rosdistro/blob/master/indigo/distribu...
These get generated into thousands of jobs: http://build.ros.org/view/All/
It would be totally unrealistic to rev job configurations by making commits into the individual repos (and in the public open source case, actually impossible— the maintainers of that community build infrastructure don't have write access to many of the repos— it's equivalent to a Debian distro using DSC files to grab tarballs).
The jobs page is not loading for me right now.
I think that in general developers and the build engineers should both have commit access. But I see your dilemma and think we need to have a conversation to see what the possibilities are.
This isn't exactly what you asked for, but we're seriously considering letting you "include" CI configuration from a central location.
You can choose to keep the build output from a failed build, which you can download as a zip file, unpack locally, and look at the reports. This user experience makes test reports mostly useless.
At the same time, they have test coverage badges, where test output is parsed to provide coverage. To me, that's really weird: You don't get proper access to test failures, but lots of work into test coverage.
(I have no knowledge of this, but I suspect that being a Ruby-shop, they have little experience with working with java projects, and therefore don't really understand how the toolchain works. This is all guesswork, though).
In the end, we cannot give up Jenkins. We need to know the result of our tests, and see the test output!
But I agree with your comment that we should extend the scope of the integration and make it work better with hosted JIRA. Our VP of Engineering is assigning someone to this task. Feel free to email stan@ company domain with a link to this comment and with what you need so we can have an engineer look into it.
Many companies use JIRA and it should work great with GitLab.
Most of our new users use only part of GitLab's functionality and we want to support that use case with extensive integrations and APIs.
The approach of including a GitHub -> GitLab API translation layer would allow GitLab to integrate with any application that can use GitHub. I've added this info to this ticket: https://gitlab.com/gitlab-org/gitlab-ce/issues/20418#note_15...
Big fan of GitLab btw, keep up the good work!
Is there functionality being built-in to integrate with Gitlab instead of being forced to use everything Gitlab has to offer?
Applies to job title, too. Yes, even at companies who are special enough to get millions of dollars from VCs.
source: "In Comments" https://news.ycombinator.com/newsguidelines.html
When you're a new startup and you're trying to race to an MVP and being default-alive, what do you really want to do? Do you really want to spend time getting all of your ALM and build pipeline and production environments up and running? Or do you want to start iterating on business features as quickly as possible?
The truth is that providing 80% of the features is going to be good enough for 80% of new companies. When companies figure out that they need one of the 20% of the features that are missing, they can then migrate to get the feature that they positively identified a need for. Putting in much more effort in the beginning for ALM/pipeline/environment features you may not even need is a form of premature optimization.
The 20% missing features can be added via API's (we're trying to add API's for everything people need https://docs.gitlab.com/ce/api/ ). Or even better, by having people contribute code to the project itself.
Isn't the Stash-Jira-Bamboo-Confluence stack a unified UX for doing all of this (source control, pull requests, deployment, docs, issue tracker, build pipeline, etc)?
It may not be a great UX, but can we at least give Atlassian credit from probably having thought about it having a great UX?
I used the entire Atlassian suite for 3 years at my first web development gig, when I started over 7 years ago. I just have all these memories of things being ugly, slow, and hard to customize. Although I had a mostly positive experience with Confluence. Bamboo was just not nice to use. BitBucket does most of the same things, but it just didn't feel nice to use. Maybe it's the UI, or the dark blue theme. Maybe it's purely psychological, like GitHub was the place where all the cool developers hang out, and BitBucket is the corporate nerd who just wants to fit in so they give away all their private repos for free. So I dumped my private, personal stuff on BitBucket, until GitLab came along.
And honestly, up until the last few months, I actually really disliked GitLab. I kind of saw them as just rip-offs who were blatantly copying GitHub, and stealing their customers. But then I realized that GitHub hasn't really done anything interesting for years and years, while GitLab is working on all these awesome features and integrating things that GitHub should have been doing years ago. So now my opinion is that GitHub had their chance, and they blew. GitLab is the new cool place to host your code.
In my case having the Atlassian all tools look the same seems to have backfired; every time I go to BitBucket it makes me think of Jira, which makes me cringe and wish I were on a different web site.
We use GitLab for both internal and client project code, including their runners to run limiting on code (for example) so web based edits/commits can be checked with the same rules we may run locally. Love it.
We use GitLab wikis to replace Confluence (which had replaced Evernote because, well, I couldn't trust them any more). We sometimes use Gollum locally because sometimes a UI around editing pages is helpful, though I sure wish prose.io connected to GitLab. I'd also like to see GitLab switch to Kramdown for Markdown rendering (there's an issue open for that already) and actually use the Gollum gem to render the pages, so that we can use the various partials that Gollum supports when it's serving up a wiki.
Once time tracking lands in GitLab, our (admittedly) basic usage of Jira can be replaced, since we're already using guest/reporter accounts for customers to report all issues on any custom dev project we do for them. We'd like to see that on-boarding process (for our clients who we intend to use Issues) go a little more smoothly, and I'm working on a written spec for proposed improvements, and I'm comfortable enough with their code base to take a whack at writing it.
If it's not dev work (e.g. a general SharePoint support contract), we just use a new HelpScout mailbox with a custom email address for the client. That email address is just an alias that forwards to the primary HelpScout mailbox, where a workflow rule moves it to the client's specific HelpScout mailbox. Nice thing is that HelpScout's Slack trigger doesn't fire until after the email is moved, so notifications also go to the channel we use to discuss that customer's issues.
If Mattermost got bundled with their hosted offering, that'd be killer.
We're using their free hosted service, and so I'm reticent to ask for these sorts of changes by opening issues unless I want to at least attempt to contribute the code for it. I'm unsure how some of that would work out in their multi-tenant environment, but I still plan to dig into it.
GitLab is good stuff. I've been deeply impressed by their agility and responsiveness to their customers. If anyone is on the fence, there's no cost to check out early all of their features.
I think we use the Gollum gem for rendering but that many of the features need to be enabled individually https://gitlab.com/gitlab-org/gitlab-ce/issues/2073
I'm very excited about the first iteration of time tracking that will hopefully land next month https://gitlab.com/gitlab-org/gitlab-ee/issues/985
We're looking into hosting Mattermost ourselves, but it will be a couple of months before we have the access controls right I think.
Thanks for all your kind words.
Please feel free to open issues anyway. We love hearing directly from users and always want to improve the product, even if you don't pay for it or contribute code. :)
The jetbrains stack VS The atlassian stack. A long standing competition between the best tools in the world.
- Support downloading the code from various repo (GIT, SVN...)
- Option to start build from scratch or keep the state from the previous build
- Integrate with your user accounts (LDAP, Google Auth...)
- Good and stable API [jenkins API is a complete mess]
- Storing and retrieving build artifacts
- Bisecting
- Running batch of builds / parametrized builds
- Graphing and statistics (build durations, build failures, ...)
- Backup system for the configuration of the CI master (must be scriptable)
- High availability: Either support a pair of CI master with active/passive, or support scaling horizontally with many active masters (this is critical for > 1000 dev companies with many many projects)
- Build in monitoring in the CI master itself (e.g. manage the space taken by projects and artifacts) with a notifications when things are going critical.
...
Is there a CI server that can't handle this? Worst case you can do a checkout from the command line.
> Option to start build from scratch or keep the state from the previous build
This should be part of the build script. "make clean build" or "make build".
>Integrate with your user accounts (LDAP, Google Auth...)
Not sure whatthe case for this is, but the authorization can be passed to the build script anyway.
> Good and stable API [jenkins API is a complete mess]
What do you need the api for?
> Storing and retrieving build artifacts
Again, better off being a feature of the build script.
> Bisecting
This is a job for humans.
>Running batch of builds / parametrized builds
Doesn't every CI server have this?
> Graphing and statistics (build durations, build failures, ...)
This is really not important.
> Backup system for the configuration of the CI master (must be scriptable)
Why is this part of the CI server and not a general backup system?
> High availability: Either support a pair of CI master with active/passive, or support scaling horizontally with many active masters (this is critical for > 1000 dev companies with many many projects)
This really isn't important. The CI server being out for a few hours will not bring down the company. If you have that many projects then there is no problem with having multiple build servers.
>Build in monitoring in the CI master itself (e.g. manage the space taken by projects and artifacts) with a notifications when things are going critical.
Like the backups, this should be part of a general OS monitoring, no need to complicate the CI server.
>Is there a CI server that can't handle this? Worst case you can do a checkout from the command line.
It's common NOW. It's been a problem for a very long time. The transition to the git era took a while and didn't happen smoothly.
I didn't check, but there may be some of the SaaS services being bound to specific repo services.
>> Option to start build from scratch or keep the state from the previous build
> This should be part of the build script. "make clean build" or "make build".
A script can't "rm -rf ." at the start of the build or it destroys itself ;)
A script could "rm -rf ." at the end of the build... except when the build fails midway, leaving the machine in an inconsistent state.
>>Integrate with your user accounts (LDAP, Google Auth...)
>Not sure whatthe case for this is, but the authorization can be passed to the build script anyway.
I am not talking about build. I am talking about people signing in to the web ui with some sort of user accounts.
>> Good and stable API [jenkins API is a complete mess]
>What do you need the api for?
To add/remove slaves programmatically. To trigger builds and get build status. Among other things.
>> Storing and retrieving build artifacts
>Again, better off being a feature of the build script.
The CI SHOULD have a feature to store artifacts or upload to ftp/ssh/s3. (other)
There is no excuse for not supporting common use case. It's better to have CI's dev program it right once, than having one million user each baking their own buggy scripts.
>> Bisecting
>This is a job for humans.
WTF. The job of a human is not to click 20 times through the interface only to enter the git hash of 20 commits.
Just gimme a button "build last N commits" or "build commit between X date and Y date"
>>Running batch of builds / parametrized builds (had matrix build in mind specifically)
>Doesn't every CI server have this?
As a matter of fact: No, they don't.
(I was also surprised to discover that in the wild).
>> Graphing and statistics (build durations, build failures, ...)
>This is really not important.
It is important. We NEED to check the evolution of the build duration.
>> Backup system for the configuration of the CI master (must be scriptable)
>Why is this part of the CI server and not a general backup system?
It is the responsibility of one's system that the system is backable.
If all the configuration of the CI is a "conf.xml". It's okay, I can just copy that file.
If all CI master is a mess of thousands of directories and files, where the master node configuration and the build configurations and the past build info are intertwined. I am forced to backup everything together, which may be 100 GB of data.
If the past builds and current configurations are referencing each other with unique GUID and numbers, in a way that ANY file missing might cause the jenkins server to not boot... I am fucked, especially so because there is no way to backup atomically 100 GB of data across 100000 files and directories while builds are running.
> This really isn't important. The CI server being out for a few hours will not bring down the company. If you have that many projects then there is no problem with having multiple build servers.
Given your arguments of all things being useless. You seem to be working in a very small company or not have worked professionally at all.
Please try to envision a real company: When the CI system breaks, there are 100 people who are interrupted and they will be partially unable to work until it is fixed.
By The Way: It's easier to maintain and manage one big system. By splitting one CI pipeline into 10 CI, you just created 9 SPOF and you just increased the failure rate dramatically.
>>Build in monitoring in the CI master itself (e.g. manage the space taken by projects and artifacts) with a notifications when things are going critical.
>Like the backups, this should be part of a general OS monitoring, no need to complicate the CI server
It is the responsibility of one's system that the system is reliable.
First) CI servers have disk space issues. I expect a sensible way to find out what projects/builds are taking and the ability to clean it up. [a monitoring system will give full disk alert, it doesn't understand the CI internal organization]
Second) If the dev are not retarded, they can put a notification for everyone to see when the disk is at 99% instead of writing the last % and crashing... only to never reboot again because it corrupted itself in the process.
Go compare keyboard shortcuts between all these services and you'll quickly change your mind.
This is very important. From the GL's PoV an integrated suite sounds great because the integration is easier and you can capture more of the value chain. But not only is it a barrier to adoption, if successful it squeezes out innovation. The whole point of open source is user benefit, right?
The integration with our own tools will always be better, because we can do things that'd be hard with external tools, but if you want to use something else for anything, it'll work with GitLab.
If that's your intent then my statement is not operative, great. But your other public statements imply (to my reading) the opposite of what you say in your comment. For example, from the announcement:
> We prefer to offer an integrated set of tools instead of a network of services or offering plugins for the following reasons:
and some comments from your colleague theunquietone: > We believe in a single solution from idea to production.
and when someone commented "Makes for a tough pitch at an established shop" he replied: "We hope to create tools and resources to make that pitch and migration easier."
It's early days so you have time to clarify/refine your story.
Come to think of it, services like Travis CI could integrate with gitlab.com easily (if they wanted to do so).
That said, we don't expect to be able to replace every single tool one might use -such as internal tools-, therefore we make it possible to integrate with GitLab easily and have an API that covers everything.
Examples of this are plenty. For instance, we build our commit status API to support our own CI system, but immediately also published the public API that allows any other CI to make use of this and offer similar integration in GitLab (in that case, you'll still have to switch between apps, which we believe can be better - hence our goal).
It was done for the right reason: to have a single place for all our work (JIRA for feature planning/sprint planning/dev+qa tracking, Confluence for documentation) but it's been hobbled because we still need github to coordinate code review + merges.
The better solution would have been to stay with Github and use Huboard or something to layer on the swimlane sprint planny stuff, but keep _all_ our info in GH Issues.
Moving entirely to Gitlab in a year or so once things have matured would be a great win for us, I'm sure.
As someone who has been doing product development their entire career, I have found there to be an economy of focus in a software company when it comes to product development. I think it's a great vision for Gitlab but it's risky because it splits their focus into like 6 beams. The major players in the market (Atlassian, GitHub) already have this vision and they are there already pretty much.
The market wants flexibility in their tools in spite of what people on HN tell you. Atlassian has tools for every lifecycle stage but people still mix and match with other tools. Github is missing a few of these stages but I'm sure they are hard at work filling in the blanks. I know people who use GitHub and JIRA together even though GitHub has it's own issue tracker.
And what I learned was that we were able to do CI and make it a better product that what was on the market. That was counterintuitive for me.
But we've since seen the benefits of integrating CI, CD, issue boards, and a docker container registry. We still have lots of work to do integrating Mattermost, Koding, Cog, Prometheus. But we're convinced we can do a better job.
If you look at what we shipped last month I hope you can see the rate of improvement https://about.gitlab.com/2016/08/22/gitlab-8-11-released/
And to make sure we have enough capacity going forward we hire remotely, use open source collaboration processes, and just raised more capital.
Microsoft did with TFS. That was enough to convince me it's a terrible idea.
I like bringing the right tool from the job rather than the massive hodgepod mega site
I like having both options so I can go with whichever option hurts less.
TFS is such a bad product that most teams at Microsoft don't use it.
You use XAML to configure builds.
The source control system still uses a locking workflow.
The issue tracker managed to be under powered and over complex at the same time.
I had to suffer through it at two previous jobs. Never again.
The only reason it still exists as a product is because of Microsofts sales force, not it's own qualities.
Unix Client is flaky and inconsistent and doesn't get much love.
I can't put a version number into a file so I can see in Prod what version the code is.
I could go on.
We have discussions about Jenkins vs Concourse, where to keep ansible vault passwords, should documentation live in Github wiki or in Confluence (apparently "tech" documentation in the form, "business" documentation in the latter - what if it's both? Who decides?), and so on.
There is something nice about being able to go to a single place and saying "OK, it's all here in this box". Github has made inroads with some of this stuff, but not quite enough. Gitlab could try and do all this, but then people will moan ("I prefer JIRA/Trello/whatever").
Most of the pain around developer/business workflow around us at the moment actually comes down to the fact that nobody has _really_ thought about providing a great unified UX for all of this.
Part of the concern is people want to be "flexible". No, dictate, just make sure what you dictate is a better solution to what people have.
If GitLab get it right, github could be a minor player (unless they keep up) in a few years time.