Taking this further, the upcoming release of Jenkins 2.0 is going to put a lot more emphasis on pipelines-as-code, where entire workflows can be defined in code, and version-controlled, as opposed to clicking everything together via the web UI.
* Plugins are useless
* Configuration drifts all over the place
* Jenkins ends up just being used as a job runner to run shell scripts
* Every repository or project implements similar but subtly different build scripts
Instead, we ended up using resque. We're a rails shop and already use resque extensively in production. Resque scales really really well, so we were pretty confident that we would be able to run builds on this sytem indefinitely without needing to split anything into separate clusters. And resque jobs are ruby, so they can be written and tested just like ruby.
As a result, we were able to standardize on just a few jobs (e.g., BuildRailsJob, GradleBuildJob, NodeBuildJob) with a few arguments for each job. In the process, we wrote a lot of really nice primitives, so if we do need a different kind of job (or a modification to an existing job) then those can be made pretty easily. On the whole, I've been extremely pleased with the resulting system.
* Reporting such as unit test reporting
* Emailing and notification
* A well known plugin system
* User security
* Master / Slave management
Really Jenkins just needs to fix the configuration drift issue.
.ci.yml like .travis.yml https://github.com/groupon/DotCi
solves every single problem you have listed.
Looks like jenkins has discoverability issues. I am curious what kind of things you tried with jenkins before writing your own build system.
First, we made all of our builds into docker containers. Under this system, slaves (we call 'em executors now) only contain very basic software - docker, git, and make to be specific. This means that our builds are entirely self contained, and we don't have to worry about, for example, messing around with RVM on executors. If the containerized build works locally, it pretty much always works on Jenkins as well.
Second, we started using the job dsl plugin to manage configuration and brought in some autoscaling and machine identity. Beyond what I do as a platform engineer to turn my projects into jenkins builds, I don't completely understand it (nor do I have to, which is a good thing!), so I'll let our DevOps guy take it from here:
https://blog.conjur.net/scaling-jenkins-with-machine-identit...
I am very bullish on Concourse.
Ultimately, we took an approach similar to Travis CI, or Gitlab CI [1], only using shell scripts since that plugs into Jenkins easily enough. Every project has a CI script and a release script in a common location relative to the project root that takes care of everything needed to take a fresh clone of a repository, run the tests, and deploy the project (depending on a few environment variables) if the tests pass.
We have 1-click operation to set up a new job in Jenkins, and it handles all the configuration based on an XML template. Everyone understands that they're not supposed to make manual tweaks to the jobs once they're set up, and a year later, things are working pretty smooth.
thanks for sharing.
To resolve that we've tried to push the shell script into a file in the repository, which Jenkins then checks out and runs, which makes it easier to maintain and faster to set up new build machines.
I've seen pipelines with hundreds of inputs (an embedded software company), others with over a dozen stages (other teams at Pivotal), both kinds with fan-in/fan-out as necessary. Today I even saw a generic pipeline that could test and build identically-structured product files in a uniform way across quite different products (Redis, Apache Geode etc).
People have written resources (the main means of extension) in Go, Ruby and Python so far. If you can put it in a Docker image and execute it, then it can be taught to behave like a Concourse resource.
So far it's working well for us and others trialling it. I am very bullish about the future of Concourse.
My first approach was as an individual, which was quite difficult, because it took me a while to work out the differences between jobs and tasks and how to lay it all out. Lots of copying and pasting from other pipelines I studied.
At this point pretty much every team in Pivotal's Cloud Foundry division is running a Concourse pipeline, including the one I belong to. What's been interesting is how each team is experimenting with patterns that Concourse makes possible.
For example, I and my peers are now turning various things into "executable documents" -- the terminology sucks at the moment. Think of all the stuff buried in READMEs and wikis and long-forgotten cron jobs. How to build that special docker image that you only update every few months. The database backup. Keeping the blue-green deploy codepath warm.
When we find another one of these, we now encode it into our pipeline. That way, if I need to find it, it's there. And if it goes bad, the pipeline definition points me at where to go looking for everything of interest.
Another pattern that is emerging for us is "enforce project invariants". For example, we have multiple repos, so multiple Gemfiles and Dockerfiles, all of them with ruby versions set. At the front of our pipeline there is a little gateway to check that these are all identical. If not, it prints a table of versions found, so again, I know instantly where to go look. We have various other little invariants that turn days of insane debugging into a few seconds of sanity-checking.
None of this is novel. What Concourse lets me do is hoist good design practices out of code and apply them to CI. The final feedback loop I rely on can itself now be checked in, broken apart according to SOLID principles and even (my colleagues in Buildpacks are pioneering this) unit tested. Some colleagues in London have designed a fully generic pipeline that can be configured at runtime to build any of the five products they manage.
tl;dr fuck yeah Concourse.
One of the things Spinnaker does is plugin Jenkins jobs as a reusable stage that is parametrized and scoped to an application deploy. This has allowed one team to move from 40 Jenkins jobs created via job dsl plugin to just 6
Could use something like this if you are using github https://github.com/groupon/DotCi
Job configurations can be version controlled and reviewed like everything else.
Also, the plugin does other optimizations like storing job/build data in a db so jenkins doesn't slow down as you create more builds/jobs. So you don't need "25 Jenkins masters".
For multijob pipelines checkout https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin
I was surprised about needing 25 different masters. On GitLab.com we have a single clustered application that handles over 1600 GitLab Runners (called build slaves in Jenkins).
Our team ended up forgoing the plugins and simply writing shell scripts (checked into our repository) that would handle various pieces of the build workflow. Our jenkins job then became:
Call script 1
Call script 2
...
At Pivotal my colleagues working on Cloud Foundry poured a lot of engineering effort into making GoCD scale across multiple teams, repos, sites and so on, and it just never worked out.
Alex Suraci wrote Concourse, dogfooded it on a project team, and now pretty much the whole of Cloud Foundry is being built with Concourse pipelines.
What are some of your thoughts on,
* How it compares to Jenkins pipeline plugin https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin
* Could it have been written a jenkins plugin instead of a whole new CI software. I like some of the features of concourse pipeline but it doesn't have support for wide range of remoting/plugins that jenkins supports.
As for writing a plugin, no, it would not have been possible. Concourse has an entirely different model of operation and needs easy access to containerisation facilities to achieve it.
Concourse doesn't really think in terms of "plugins". What you become accustomed to is wondering "is there a resource type for this?".
Right now I work on a software repo with a moderately complex API for uploading final binaries and releasing them to clients to download. Instead of telling people to write scripts or install a plugin, I can point them to the resource that another team has written.
"Just add the resource". Released software is now just a stream of events, no different from git commits, S3 files, points in time, Tracker stories etc etc. Every resource has the same interface so it makes it possible to click together stuff into clever combinations, rather than lashing together awkwardly and hoping it'll work.
We are currently in a state of "big ball of plugins and configuration". A bunch of plugins have been installed, and lots of manual configuration has been put into jobs so that everybody has what they need to build their software. It has led to Jenkins being a "do everything" workflow system. The easy path that Jenkins provides, to me, seems like the wrong one - it makes it easy to just stuff everything in there because it "can" do it. This seems to leads to tons of copy/paste, drift, all types of different work being represented, and it is starting to become unmanageable.
Have others seen this happen when using Jenkins? How have you dealt with it?