Gradual rollout also helps (whenever possible), especially with stuff that has to work on many hardware platforms etc, so you can roll out only a few to a new hardware platform, see that it works, then roll out the rest.