- Automated tests that run on push to any branch.
- Rolling deploys (no downtime) from any branch.
- Exception reporting system (Exceptional, Hoptoad, homegrown, whatever).
- Twitter search feed.
Push to a topic branch, wait for the tests to come in green, deploy from topic branch, watch exception notification and twitter. If everything looks okay after 10 minutes, push to master. If exceptions or twitter blows up, redeploy master.
You should be able to setup the whole system in about the same time as it takes to develop a majorish product feature.
There is also a Plugin (https://wiki.jenkins-ci.org/display/JENKINS/Github+Plugin) which should handle this automatically.
Its not overly hard to setup, I just added git repos to the list. I'm still "testing" it locally on my workstation before we start using it in all of our environments.
(Most surges are negative, eg. "Man, <YourService> is down AGAIN, this sucks.")
I think doing CD as early as possible in the project is very worthwhile: It forces you develop high confidence in your app from very early on. When committing your work means that users will be using your code in a few minutes time, you get in the habit of making sure that you have appropriate test coverage for the code you're committing. This also makes it more difficult to take shortcuts or "I'll add tests around this tomorrow".
If you don't get in the habit of CD early on, you're risking developing bad habits such as manual testing or hard-to-automate release steps which makes it progressively harder to start doing CD.
I run it on my personal computer. I also set it up at our company, where I'd say the active user base is around 10 people. It's useful in both cases.
Jenkins has plugins for everything, and soon you'll be able to develop plugins in Ruby, and without maven, so if you don't have what you need you'll be able to add it.
If you can handle doing it manual, do it. There will be a point where you are growing too much and too fast, and you loose too much time deploying, then look for implementing CI.
A lot of people will tell you to use CI from the start. And our startup has been using CI since the beginning, but there was no apparent benefit in the start. It was only 1 full time developer, and a second part time.
Right now we couldn't live without it, but this is because our applications has grown considerably, and the speed at what we are implementing our ideas is fast. Loosing time doing Deployment and running tests manually would be a great loss of time and of flow.
CI is only worth it, if there is a benefit for your project.
Jenkins has an account on said XMPP server (our AD logins actually) and is thus able to send messages to the developers that are signed on. We also have a "conference room" set up on XMPP that every developer is logged in to most of the time, and Jenkins is available there to run commands, so you can for instance say !build android-client and it will start building the android client, test it in the emulator and report results to the chat room.
I'm guessing that this is a pretty powerful computer to run the emulator on?
We have a gitorious server that also has a jenkins account (jenkins connects over SSH). It is running within a VMWare Virtual Machine with access to 4 cores running at 2.4 Ghz, and 14 GB's of ram. Whenever jenkins notices a change in git (using polling, every 5 minutes, set up for now, eventually I'll get around to doing push notifications of some sort ...) it pulls down the latest source code, and starts up the Google Android emulator (basically qemu for the ARM platform with the devices as an Android phone would have them), once that is started up it compiles the source code (java) and using adb installs it on the phone, runs the test suite, and reports any errors back, Jenkins then shuts down the emulator.
It isn't extremely fast, starting up the emulator takes about 50 seconds or so, and I am trying to get a faster server to use as a jenkins build slave, but for now it works wonderfully. Everything I mentioned is completely managed by jenkins.
When does setting up something like Jenkins CI become worth it? When you have 2 people? 10 people? 100 people?
Is there tons of custom code to set it all up with Github or whatever deployment scripts exist? What CI systems are dominant? (I'm mostly curious about Ruby-centric ones, but don't really want to bias responses.)