One really cool reading about reverse engineering obfuscated binaries and protocols, getting around anti-debugger code, etc is this talk from Blackhat about reverse-engineering Skype:

https://www.blackhat.com/presentations/bh-europe-06/bh-eu-06...

Gives pretty good insight into what steps go into this kind of work.