Script your build so that you can run it with a single command on any build server or any developer machine. Then have the build server invoke said scripts (with arguments for any build-server specific paths etc).
Having spent a week fixing problems with inter-plugin dependencies I do agree with your point - Jenkins just isn't very good, like a lot of the free software from that generation (FitNess etc).
Glad to see jenkins is moving towards a travis-ci ".travis.yml" kind of format, but I'm not hopeful to see it catch up to anywhere near TeamCity, even Bamboo in user or developer UX.
For multijob pipelines checkout https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin
Jenkins 2.0 will provide support for build chains via the pipeline plugin which will be included as a core plugin.
Access control can be configured quite nicely with the Role Strategy Plugin: https://wiki.jenkins-ci.org/display/JENKINS/Role+Strategy+Pl... + AD/LDAP integration.
Reporting automation is covered by the pipeline plugin. Custom reporting is quite easy to implement on top of Jenkins, its APIs cover almost everything you need.
What about cascade builds? The pipeline can't be handled in your build script.
What about reporting? (list of compiler warnings in the build? List of tests that failed? In general extracting information from the log)
What about efficient email alert? (with blame list, list of changes, etc.)
What about IRC reporting?
Jenkins is not very good (and written in Java :() but the other available free alternatives are just not better.
You can do a heck of a lot more in scripts; you've got a full language with all of the libraries it comes with in order to do stuff.
We've build our current build pipeline with Python scripts:
0. Bootstrap the build environment (installs everything you need to build). 1. Build projects order, order is derived from the intermodule dependency graph. 2. Execute unit test sets based on build parameter + save results in output. 3. Execute integration tests based on build parameter + save results in output. 4. Build distribution packages + save in output. 5. Deploy distribution packages to server based on parameters. 6. Run UI tests against the deployed packages + save results in output. 7. Publish the bits if they're OK.
Most of these steps are in different scripts, which can be called either from orchestration scripts or one-for-one in Jenkins.
All we have Jenkins do is invoke the script(s) then process and report the results.
We've basically got it wrong with the design of our CI tools; they do too much. They should just provide scheduling, invocation, reporting and dashboards over the build process. Expose that as an API and let scripts integrate with it.
In any project, if the build fails, you want the latest commit automatically reverted.
Now, do you just copy and paste to each script, or do you have a chain of build processes and logic in the build system?
There is a separation of concern: the CI server logic is a refactoring of things that are common to any project. If your project have scripts that sends email on build failure, I think you're doing something wrong.
You mention at some point "orchestration scripts", this is exactly what Jenkins is supposed to provide help with: orchestration.
The important thing IMO is that the boundaries of responsibilities are clearly defined between what a project does and what the orchestration does.
Too often I see engineers bias heavily toward inexpensive, free/cheap, or home-built solutions simply because they personally don't think things should cost money, when the understated anchoring costs of building and maintaining, plus the opportunity cost of valuable people time, might shift the toward other alternatives if a wider business context is taken into consideration.
If we're talking about hosted solutions, I agree it's mostly silly to refuse to pay... unless we're talking about open source projects, where many CI vendors do have a free tier...
Anyway, I also see an opposite bias toward costly hosted proprietary solutions and a refusal to even think about how one could accomplish one's goal with, say, a couple of shell scripts and cron jobs—or a self-hosted open source system.
That bias can be very sensible, but it can also be a hindrance, especially when it turns out that the proprietary solutions aren't quite what you want anyway, but since you're opposed to writing your own scripts, or running a more flexible system on your own servers, you settle for a suboptimal setup...
This bias I'm talking about is also a way of undervaluing something. It's saying "I'll pay $99/month for some hosted thing, but I'll never spend a week of engineer effort on really getting it right."
Now our build and test suite is just a script each and it can pretty much on on any CI server.
If you're looking at Jenkins purely as a CI then absolutely, it's overkill. I've seen Jenkins used as a distributed cron replacement, CI server, pingdom replacement, deployer and even as a means to give non-programmers permission controlled access to execute certain jobs via parameterized builds. Just having granular control to decide how to report on cron failures from a central location is a huge help.
It's ability to combine job execution, detailed result tracking, scheduling, scripting, SSH and failure reporting from a central location with flexible permissions makes it an extremely useful tool.
The extensions that the Cloudbees folks have put together are pretty sharp too.
This is obscenely bad advice. We run thousands of builds/day and none of the configuration is maintained inside jenkins. All of our job configurations resides in repository alongside all the code.
$12K/month for a much (I mean, it's like going from cvs to git) better solution in that situation should be seriously evaluated; perhaps you could give every team in that org their own dashboards and eliminate/distribute the whole release engeering org. $12K/year, which is the actual TeamCity pricing, is a no-brainer, which unfortunately most teams are way too cheap to pay for, headcount looks so much better. Fortunately at least cloud based CIs are starting to seriously take off, hopefully I never have to deal with a home-grown Jenkins deployment again.
So in my obviously biased opinion, if your team is big enough, either get an actual build/release engineer/devops guy or appoint one of the devs as an owner for this, otherwise you won't get the best results.
Also, arbitrary shell scripts, (as usual) provide duct tape for everything else as needed.
However, the new Jenkins 2.0 fixes many of the issues you've described. Build pipelines are a major new feature, as well as the integration of a "Jenkinsfile" which allows configuration of jobs via a proprietary DSL.
This makes no sense to me. Only how my project is built should be checked into the project's repo, not the whole CI/CD deployment pipeline! If I start deploying to Azure instead of AWS I need to commit that change to my project? It's nonsense.
Come on now, we don't need to go to these contortions just to avoid saying that Jenkins is doing it wrong.
You could indeed just use the Jenkinsfile for building a project. You can also specify the DSL config directly in a job, e.g. in a separate deployment job.
What would be the right place to define the deployment config?
This Jenkinsfile is their solution to the problem. I'm saying it's not a very well thought-out solution, because now they're forcing us to put this thing that's not part of the project in the project, if we want it version-controlled.
There is something amiss in the tech world these days. Too much esoterica and too many specialized tools, too many points of failure and dependencies, and too much duct tape. But, mostly, too much reinvention for frequently incremental "gains".
It seems normal, but it's really a mess. Environments are overly complex and it's a wonder anything works half the time.
At some point we will completely jump the shark. From there, my bet is there'll be a powerful movement towards simplicity.
/rant
We have a custom wrapper around the jenkins-job-dsl that encodes our conventions and templates to generate several hundred jobs, and it works very well for our medium-scale instance.
Works just fine.
As much as Jenkins sucks sometimes, it's what many big shops use for many things. There is tons of community around it. It works, it's battle-tested, and it's been that way for years. Anything else will have a hard time catching up for as many use-cases as Jenkins currently solves for people.
If you hate a part of it, fix it and open a PR :)
Its simply too difficult to use by a laymen developer or sysadmin when accommodating complex buildchains, which only grow more complex over time in most software businesses.
If you're going to program your CI with a web administrative interface, and provide an easy to navigate interface for users, expand with build metrics and external systems, I suggest TeamCity over any others.
Though jenkins is free, my experience is maintenance teams of 2 or more people with tribal knowledge develop around the tool that far outweigh the license cost of TeamCity, which solve many of these homegrown engineered problems as part of the product cost.
If you have a strong developer team, I would suggest using something like python buildbot and programming your CI end-end in a single unified language, much better than shoe-horning plugins and groovy code inside your jenkins textarea fields!
That said, all CI systems are just software, and any requirement can be accommodated with sufficient amount of additional software, forks, or changes. Any will do.