For keeping track of os-level packages and the like, I find Puppet much better than something I could build myself. It doesn't just take care of the initial setup, but also handles incremental changes. And it does so in a largely declarative language. It all guarantees that the entire installation is documented and replicable.
Guaranteeing the entire installation is "documented and replicable" is huge. Chef/Puppet can become a large part of you disaster recovery plan and increases the "bus number" in ops.
The value in this is immediately apparent after the first time you have had to rebuild a server that you inherited.
I used slack[1] for a short while, but then started my own configuration management framework for Perl[2]. I wanted to have templating and better reporting, and also had hopes of keeping a very slim rulesest -- forcing any extraneous functions to per-instance (bash|perl|python|.*) scripts. I need to get back to developing and testing it though.
The reason is that by encoding your configuration in a metalanguage such as Puppet's, you have described your environment independent of the actual box. If the box goes down and you need to swap in a new one, you just run Puppet again, and it will magically turn into the old box.
If you need to replace the box with faster hardware, for example, just run Puppet on the new box, and it will (if you have done your job properly) become identical to the new one. Suddenly need 10 boxes to handle a load spike? Just boot up 10 cloud VMs and point them at the Puppet server.
Since these systems are modular and lightweight, you can easily describe hundreds of vastly different boxes using a single configuration.
If you've done your job right with the shell script, the same will be true. The problem is that if you have more than N nodes, "doing the job right" becomes more difficult. However N is quite obviously greater than one.
Why? I prefer to simply write my own bash deployment scripts. Seems easier.