Absolutely, tasks/system commands should always be stored in VCS as a shell script, so that feature branches can exercise changes in the task itself. Unfortunately theres the other 90% to worry about, and thats where the highest complexity is, such as configuring a build chains, access control, reporting, etc.

Glad to see jenkins is moving towards a travis-ci ".travis.yml" kind of format, but I'm not hopeful to see it catch up to anywhere near TeamCity, even Bamboo in user or developer UX.

Could use something like this if you are using github https://github.com/groupon/DotCi Job configurations can be version controlled and reviewed like everything else.

For multijob pipelines checkout https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin

JJB is also a nice tool that seems like it allows similar functionality (I haven't worked with DotCi, so I can't say how they compare)

http://docs.openstack.org/infra/jenkins-job-builder/

Well...

Jenkins 2.0 will provide support for build chains via the pipeline plugin which will be included as a core plugin.

Access control can be configured quite nicely with the Role Strategy Plugin: https://wiki.jenkins-ci.org/display/JENKINS/Role+Strategy+Pl... + AD/LDAP integration.

Reporting automation is covered by the pipeline plugin. Custom reporting is quite easy to implement on top of Jenkins, its APIs cover almost everything you need.

You have simple problems to suggest simple solutions. CI logic can gets complicated:

What about cascade builds? The pipeline can't be handled in your build script.

What about reporting? (list of compiler warnings in the build? List of tests that failed? In general extracting information from the log)

What about efficient email alert? (with blame list, list of changes, etc.)

What about IRC reporting?

Jenkins is not very good (and written in Java :() but the other available free alternatives are just not better.

Huh? I really don't understand your argument. Jenkins is better than scripts because you can do more in Jenkins than in scripts? O_O

You can do a heck of a lot more in scripts; you've got a full language with all of the libraries it comes with in order to do stuff.

We've build our current build pipeline with Python scripts:

0. Bootstrap the build environment (installs everything you need to build). 1. Build projects order, order is derived from the intermodule dependency graph. 2. Execute unit test sets based on build parameter + save results in output. 3. Execute integration tests based on build parameter + save results in output. 4. Build distribution packages + save in output. 5. Deploy distribution packages to server based on parameters. 6. Run UI tests against the deployed packages + save results in output. 7. Publish the bits if they're OK.

Most of these steps are in different scripts, which can be called either from orchestration scripts or one-for-one in Jenkins.

All we have Jenkins do is invoke the script(s) then process and report the results.

We've basically got it wrong with the design of our CI tools; they do too much. They should just provide scheduling, invocation, reporting and dashboards over the build process. Expose that as an API and let scripts integrate with it.

So, let's say you have 8 projects. In each of the 8 projects, you want to run the same static analysis tool. If the violations go up, you want an email sent out. If any critical errors are there, you want the build failed.

In any project, if the build fails, you want the latest commit automatically reverted.

Now, do you just copy and paste to each script, or do you have a chain of build processes and logic in the build system?

I was basically answering "If you're putting lots of logic into ANY CI server you're doing it wrong!".

There is a separation of concern: the CI server logic is a refactoring of things that are common to any project. If your project have scripts that sends email on build failure, I think you're doing something wrong.

You mention at some point "orchestration scripts", this is exactly what Jenkins is supposed to provide help with: orchestration.

The important thing IMO is that the boundaries of responsibilities are clearly defined between what a project does and what the orchestration does.

This is a critical component of releasing production software. I would not expect to see "free" in the conversation here.

There are good reasons to prefer free and open systems, even for "production" software. Those reasons are part of why people choose to use GNU/Linux, nginx, and all the rest of the open infrastructure.

It was not clear whether it was free as in beer or speech. I do understand there are benefits to using open software.

Too often I see engineers bias heavily toward inexpensive, free/cheap, or home-built solutions simply because they personally don't think things should cost money, when the understated anchoring costs of building and maintaining, plus the opportunity cost of valuable people time, might shift the toward other alternatives if a wider business context is taken into consideration.

Yeah, I parsed them as wanting a free system as in one that's open source, but I could be wrong.

If we're talking about hosted solutions, I agree it's mostly silly to refuse to pay... unless we're talking about open source projects, where many CI vendors do have a free tier...

Anyway, I also see an opposite bias toward costly hosted proprietary solutions and a refusal to even think about how one could accomplish one's goal with, say, a couple of shell scripts and cron jobs—or a self-hosted open source system.

That bias can be very sensible, but it can also be a hindrance, especially when it turns out that the proprietary solutions aren't quite what you want anyway, but since you're opposed to writing your own scripts, or running a more flexible system on your own servers, you settle for a suboptimal setup...

This bias I'm talking about is also a way of undervaluing something. It's saying "I'll pay $99/month for some hosted thing, but I'll never spend a week of engineer effort on really getting it right."

It's never the cost for me, it's the mutability, oos I can hack on, and make it do exactly what I want it to do, proprietary code I only ever seem to get somewhat close but never exactly there.

We made the same mistake with Anthill Pro a long time ago. The plugins were of various quality and caused us a lot of headache.

Now our build and test suite is just a script each and it can pretty much on on any CI server.