Despite everything you said being true, it's still one of the few open-source, free solutions in the mobile build space.

I'm wondering if Buildkite or something newer is comparable today, but for a long time, Jenkins was one of the only non-custom ways of building an in-house iOS/Android build system.

At Yahoo (Verizon Media), the CI/CD team started with Jenkins, then built a UI to manage all the instances, then wrote their own engine that replaced Jenkins entirely. It’s a good product:

http://screwdriver.cd/

The problem is that website presents me almost zero information about the product.

Is there something about mobile builds that makes the standard Jenkins competitors like GitLab, Drone etc. unsuitable?

Even hosted Gitlab (gitlab.com) lets you bring your own runner to use whatever arch you need to run the jobs on.

I’m sure I’m missing something here though.

Large cross-platform native builds generally, not just mobile, kinda suck on the usual cloud solutions. The cloud stuff's amazing, performs well-enough, and is easy to configure if your workflow is... containers, and that's all. It gets rough in a hurry once you're much outside that, and especially with large (=long build time) apps. Many can accommodate native builds, but for anything non-tiny you'll experience slow builds or go over plan limits (and see spending skyrocket) in a hurry. Anything that requires large assets at build-time, or anything else for which it's nice to have persistent state between builds, is right out. You'll end up spending a bunch of time (=money) re-configuring your build to fit the tools, so performance is merely bad and not astonishingly, uselessly bad.

You're also more likely, with native builds on cloud CI, to run into "yeah, we support that! (I mean... we shipped it, anyway, pay no attention to all the bugs and rough edges)".

The thing is, none of that is true. Even for the cloud hosted providers (Gitlab at least) you can install your own native runner and connect it to your CI system. Then run any weird combination of commands you want.

I've not used buildkite (we don't use git), but we're moderately happy with TeamCity. It's definitely not perfect, but it's orders of magnitude better than Jenkins.

TeamCity is expensive and when we evaluated it we hit a significant bug that they couldn't fix (if you rebuild a snapshot build from early in a long chain of snapshot dependencies, it would run a quadratic number of rebuilds rather than rebuilding everything once in topological order as Jenkins does).

I've found TeamCity's pricing to be pretty reasonable; It's free to start, $300 per extra agent after 3 agents, or $2500/year for support with 5 build agents (which is where we're at right now). It's also 50% off for startups. Compared to any other bring-your-own-agent solution, it's very reasonable.

The dependency bug sounds rough; have you got a youtrack link to it by any chance?

> I've found TeamCity's pricing to be pretty reasonable

Well, compared to Jenkins it's very expensive :). I guess the cheaper options may be newer, or maybe that employer ran the evaluation in a dysfunctional way (wouldn't be the first time). All I can say is that was a consideration for us.

> The dependency bug sounds rough; have you got a youtrack link to it by any chance?

No, this was three jobs ago and was all handled verbally with their representative (who gave off a pretty unprofessional vibe, frankly, so may not have recorded it in youtrack even if that's their policy). If you want to test it out, set up something like 10 maven projects with SNAPSHOT versions all depending on each other, and set to rebuild on commit or when their dependencies change: B depends on A, C depends on B (and so implicitly on A), D depends on C (and so implicitly A and B), etc.. And then commit a change to A and see what happens. Whether you think this is a good way to set up your projects or not, it's what we were doing at that place, and Jenkins handles it fine (it will rebuild A, B, C, ... in order) whereas TeamCity (at least when we were evaluating it) will rebuild A, then rebuild B-L, then rebuild C-L because B has changed, and so on.

It's even better than that, because you can pay just once and stay on the same version forever, as long as you don't need new features and support. And JetBrains gives 50% discount for license renewals, which makes it not $2500 but $1250/year – I believe this applies to all JetBrains products, not only TeamCity.

I could be wrong but isn't Jenkins at its core just "wait for a job trigger (git push/commit to remote origin), then run some shell scripts"?

Not really. Jenkins allows a very fine control over jobs and steps orchestration. I realize it might appear as a dated glorified cron, but it's really not. There's nothing you cannot do with it, which is more than I can say about any other CI system I used.

To throw my hat into the ring: I've found the best "runnable jobs" / "digraph of jobs" system I've ever used to be Gitlabs built in CI system. The UI is nice, it's got tight integration into our existing code workflows meaning there's a close-but-not-too-close association between code and jobs meaning it's easy to say that "the code and the jobs running that code live 'in the same place'" if you want that, and it still provides all the pipeline features you'd want if you want to use your CI/jobs as a cornerstone of your infra; features like the ability to schedule pipelines, to trigger pipelines via other pipelines, remote triggering of pipelines via cURL request, pipeline DAGs for a 'make-like' build experience, etc.

I've absolutely loved using Gitlab CI for the last several years, and I highly recommend it.

My experience is the opposite. Gitlab has a lot of features but they are all half-baked. Ever time I try out a feature, I feel like I get bit for doing so.

A couple of cases

- At first child pipelines couldn't access artifacts from the parent job. Now you can opt-in to it but nothing enforces sequencing

- Some features only work with "needs" (getting parent pipeline artifacts) while others only work without it (depending on an entire stage)

- When a job you "needs" doesn't exist, instead of having a sane behavior (e.g. skip yourself), the pipeline errors. Correctly propagating and maintaining all of the rules correctly in a moderately complex pipeline is ... difficult.

> At first child pipelines couldn't access artifacts from the parent job

Why would you use the tool like this? Make jobs / blocks that generate artifacts, and then chain them together into a workflow so there isn't hierarchical issues.

Now, maybe I am working in a simplistic world, but: the only time I have parent / child relationships is when a repo or service is built and another pipeline is invoked to run integration tests or deployments. What situations have lead you to needing to pass artifacts between pipelines and not job stages?

> When a job you "needs" doesn't exist, instead of having a sane behavior ... the pipeline errors

You and I may work in very different worlds, but I personally would be quite offended if a dependency graph workflow executor saw a node with an explicitly specified dependency missing and proceeded to NOT error.

I think there are missing concepts and features in gitlab. And the advertised features don’t really match reality. But it’s still way better than Jenkins, as you can work around any issues and it’s clear how it works. It’s just a yaml file.

We (Cronitor) are seriously considering diving in here. People already install Cronitor to remotely monitor their background jobs, we have a lot of the UI and data platform challenges solved, and we are building a control plane to allow you to securely invoke your jobs remotely (via polling). I'm not totally convinced we should go this path vs better and deeper monitoring and metrics capabilities, but we're contemplating it because, honestly, Jenkins needs to be replaced.

(Readers, if that sounds interesting, and you have some Kubernetes and Golang experience, we are hiring!)

I hate jenkins groovy so much

Yeah, Jenkins can be really unpleasant. Groovy as a language isn't so bad - I just don't use it anywhere else. So... like a few other languages, I have to look things up every time I need to mess with it.

What are good alternatives?

GitHub Actions and GitLab.

I personally found both of those to be paradigm shifts (I went from Hudson to Jenkins to Travis CI to GitLab to GitHub Actions). Travis CI introduced self-serve test infrastructure configuration-as-code, GitLab introduced composability and API integration, GHA introduced many more API features (like various kinds of event sources) and better composability. Each transition also took more hosting headaches and compatibility issues off my hands.

+1 to this. Travis, Circle and Jenkins are entirely in the rear view in this area for me now. My only concern about Actions has become that I use it too much.

I have very little experience with GitHub actions but it seems much more limited and even worst it ties you to a single vendor

We run everything through TeamCity. Not going to claim its the best tool in the world but it's certainly the best I've used, and it's fairly straightforward to work with!

We use TeamCity here as well. Pretty limited in the options we have since we have most of our stuff in Perforce.

I definitely prefer TC over Jenkins though. The Jetbrains support for TC is also really good and they have been helpful whenever we contacted them or opened an issue on their public tracker.

TeamCity works well but can be a bit expensive.

It has a Kotlin DSL that is quite powerful and flexible, but of course you need to know your way around Java to make the most of it.

Custom plugins is also an option if you need something special that is not provided out of the box.

Even though I have moved on to Azure DevOps Services, it's still good to see that JetBrains have fixed a 10 year old feature request.

https://youtrack.jetbrains.com/issue/TW-17939

I would rather use Jenkins for all of eternity than use TeamCity again. TeamCity is probably the worst CI/CD product out there.

Can you give some examples?

Its been a while, but basically horrendous UI/UX, unintuitive way to create jobs and manage variables, things like that. I don't know if it was completely rewritten in the last 2 years but it was dogshit awful before 2019.

Concourse: https://concourse-ci.org/

It takes some getting used to, but I really got to love its concepts for resources/inputs/outputs and how they work together to get you actually reproducible builds.

I must admit that I got put off by the lack of abstractions for the common use cases. The resources/input/output model is so primitive that it almost feels like programming a Turing machine.

Edit: Took a look again: The situation might have gotten better since I last played with concourse

It doesn't parse tests. It doesn't track test success/failure/runtime. Because it doesn't parse tests, it can't show you the output from only the failing tests. The UI is horrid. It's so far inferior to almost all it's competition, I have no idea why anyone would use it.

Can you explain a bit more what do you mean by "it doesn't parse tests"?

Concourse explicitly wants to not know anything about the outputs of your jobs. Most of the other ci products will have _some_ method (integration with common test runners, markers in output) to figure out how many tests you've run and display them as a collapsible outline, or something similar.

Because the ci system is aware of your tests, and perhaps a filename/test description, it can keep stats on them over time.

This is an ancient version of teamcity [0], but it shows how it knows how many tests were run, how many passed, and that there's one "new" failure in this run vs previous runs.

This lets it tell you which tests are slow, which are flaky, all sorts of history of those wrt to state of the associated repo, etc.

Not having these sorts of things once you have them is like flying blind.

[0] https://i.imgur.com/ywB7u11.png

Really love the expressivity of the config syntax and the domain model in Concourse.

Seems to be a bit harder to integrate with other systems (like GH check results), and alas the community is pretty small (which probably factors into the first point).

Zuul CI, it works great for both small scale and scales very easily. It's the CI developed for openstack but it's also used by BMW and Volvo Cars for example. https://zuul-ci.org/

Bamboo from Atlassian is really good IMO but obviously not free.

If you are Atlassian shop, then yes. Otherwise, don't even bother

I like Drone, and run it on my own systems. It integrates nicely with Gitea.

Links for the lazy:

https://www.drone.io/

https://github.com/drone/drone

+1 for drone.

Note that it's intended for dockerized deploys, so it won't fit if you're not deploying containers.

That’s not true. It runs whatever you want, inside a container with your code. You can deploy from within that container however you like. You can run tests that just pass/fail, and throw away the container result. You can ssh or rsync things from within the container to other remote hosts, and throw away the CI container. It uses containers to ensure build consistency, but you don’t have to use them to deploy containers just because your jobs run in them.

Oh thanks for the correction!

Drone Enterprise is fantastic. Rough around the edges and the docs are really lacking, but it's just so god damn simple and effective.

Gitlab.

Both Gitlab and Bitbucket Pipelines fall under the "sufficient" category in my book. I've used both in large productions deploys and, they for the most part, get the job done. Gitlab has moments where it doesn't work - and one needs to go check the status page... and at times report an outage. But honestly, I still place it in higher regards than Jenkins.

Yes gitlab so so much nicer to run on than Jenkins.

Sure there is something missing for more elaborate build pipelines, but not enough missing to block you.

The best think about gitlab vs Jenkins is that there are no plugins. So it’s very clean what is gitlab and what is your config of it.

Whereas Jenkins has so many plugins and config for the plugins that it’s hard to say where Jenkins ends and your build pipelines begin.

Oh no my username is a an actual product :/

Though this looks pretty awesome!

Buildbot as a project has been around since 2003... with a username like that I am surprised you hadn't heard of it before!

Buildbot was the first CI I ever deployed at a company a long time ago. It came before Jenkins!