Really there's nothing stopping you from automating "some" machines and later pushing that out to the rest of your hosts. (Or just setting up trivial automation "configure NTP", "Disable root logins", "Append 'steve' to sudoers" - and keeping doing the rest by hand until you're confident.)
FWIW I use my toy configuration system, slaughter:
Ansible looks great. Much lighter weight but it can still scale up.
Puppet and chef would both benefit by supporting single run light weight modes of usage.
I use it to manage just 5 servers, with a Fabric script that rsyncs the manifests up to the servers then runs `puppet apply` to apply the changes.
and I see here I have capfiles and a system to bootstrap a new server. way too much stuff for this scale. (4 servers)
1) Many people seem to be recommending Puppet / Chef. How many servers or installs do you need before this is a good ROI? (Over using odd bash scripts or cPanel/WHM)
2) Am I right in thinking kernel updates don't get applied until the server is rebooted? If so, how / when do you manage this?