we may or may not have forgotten to rsync that binary to a new web server we started up.

you might want to consider using git as your deployment model.

Over the past year I switched to using git to deploy almost all of my apps & services and it is just so much better than what I was doing before (various cobbled together crap).

I even have git repos for things like my servers' .bashrc file ... you'd be surprised how useful that is.

Yes there are tools like puppet & chef and whatnot, but I think of those as better suited to dependencies than your actual code with its higher push frequency.

[UPDATE] might as well talk about how we do this:

In order to use "git push" from a development box to push to remote servers (rather than having them pull from a repo, which requires keeping a git repo accessible from your production boxes, which might not be convenient), you need two things:

1. On the server, run this:

  git config receive.denyCurrentBranch ignore

Otherwise git will refuse your pushes b/c the receiving end is already on that branch

2. On the server, write a .git/hooks/post-receive and make it executable. Its contents should be:

  #!/bin/sh
  # NB: hooks execute in the .git dir (not the repo root)
  # and GIT_DIR is set to '.'; either one prevents "git checkout" from working
  unset GIT_DIR
  cd ..
  git checkout -f # force checkout the current branch

See http://serverfault.com/questions/107608/git-post-receive-hoo... if you are curious about the GIT_DIR business.