I've spoken to multiple companies about how they do builds, and it's incredible (to me) how many of them use Jenkins. In the end, almost everyone ends up with the kinds of problems you can see on this thread:

  * Plugins are useless
  * Configuration drifts all over the place
  * Jenkins ends up just being used as a job runner to run shell scripts
  * Every repository or project implements similar but subtly different build scripts

When we faced this exact problem while writing Deployboard (https://www.youtube.com/watch?v=tgmJa7FciDg, especially starting at around 18:45), specifically the build system there. I explicitly wanted to avoid having engineers write shell scripts. People don't really know how to write them properly, nobody ever tests them, and they're just not really treated the same way you treat "real code".

Instead, we ended up using resque. We're a rails shop and already use resque extensively in production. Resque scales really really well, so we were pretty confident that we would be able to run builds on this sytem indefinitely without needing to split anything into separate clusters. And resque jobs are ruby, so they can be written and tested just like ruby.

As a result, we were able to standardize on just a few jobs (e.g., BuildRailsJob, GradleBuildJob, NodeBuildJob) with a few arguments for each job. In the process, we wrote a lot of really nice primitives, so if we do need a different kind of job (or a modification to an existing job) then those can be made pretty easily. On the whole, I've been extremely pleased with the resulting system.