You are right, it is a pretty confusing process. Here is the best docs I could find on the topic. You basically setup your domain first with Google Apps for your domain. Then you link the Google Apps for your domain account with your Google App Engine project. The additional issue is that you can't map naked domains (http://domain.com) to Google App Engine directly. Instead you have to do that within your DNS provider (I set it up in GoDaddy).

http://knol.google.com/k/larry-h/google-apps-discussion-grou...