My workplace has primarily switched all net new ci/cd workflow from Jenkins to Drone (https://github.com/drone/drone - Drone is a Continuous Delivery system built on container technology.). Having managed masters with 20,000 + configured jobs I agree that Jenkins is a very sore spot for our entire pipeline. We’ve had greater success with increasing the number of masters (one per team or application) and using the kubernetes plugin for slaves. However, most of our pipeline complaints have been mitigated by getting teams to migrate from Jenkins to Drone.

While I don’t think Drone currently supports k8s job, it is very lightweight in comparison to Jenkins. Most of our developers would agree that it isn’t a perfect product, but it has reduced friction points by a large amount.

Interesting war story. We have a couple of master instances that go down all the time because of several thousand jobs running. Gonna ask our devops what they are cooking up to resolve and compare to this solution.

Allow me to shill for Concourse[0]. It's oriented around 'tasks' composed into 'pipelines'. Each task either fails or succeeds with the yaml pipeline definition providing hooks into either state to proc other tasks. A task is a yaml definition that invokes a bash script. Each task runs inside a preconfigured docker container for full reproducibility.

It's declarative and everything needed to bring up a pipeline can be stored in git alongside the software it supports. For an example pipeline definition, check out the one for bosh-gcscli[1] and its accompanying tasks[2].

Now, there is the downside that upkeep is a hassle because it requires buying into the BOSH ecosystem to deploy Concourse. That pays for itself by being explicitly anti-brittle, with support for migrations and canary rollouts builtin.

Disclaimer: I worked with Concourse at my previous position but currently have no financial stake in its success.

[0] https://concourse.ci/ [1] https://github.com/cloudfoundry/bosh-gcscli/blob/c908176/ci/... [2] https://github.com/cloudfoundry/bosh-gcscli/tree/30316d6/ci/...

We use Gitlab's built in CI and it's been a dream. Pipelines, Docker, artifacts, caching, coverage, in-repo config, Windows/macOS/Linux.

You're not the first I've heard going this route... +1

That matches my experience. Jenkins is built out of a spiderweb of plugins and that makes it very fragile. Though, I did find I could mitigate the problem.

* only use Jenkins stable releases

* snapshot the system before updating or installing anything

* make regular backups of your Jenkins server

* never install a plugin unless it's actually neccessary

* always update Jenkins before installing any new plugins as otherwise you may end up with an untested mishmash of old and new plugins

I like having control over my CI, but it does seem a lot harder than it should be.

I replaced it with saltstack. I had so many things not working I thought it was me just being dumb or not spending enough time learning to use it. A few salt states on the end of a git hook did everything, auto deployment, testing, staging, monitoring. I am probably missing something about the awsome power of jenkins but know firms still running my saltstack CI/CD systems years after I left. Weird.

We use JetBrains’ TeamCity CI.

It’s really nice, but expensive if you exceed the limits for the free version (3 agents and 100 build configurations). Luckily we are a fair way off that so can go along fine without having to pay yet.