This is only true if your correspondents are all using the same service. If you use gmail then google knows the entirety of your email network. If you host your own mail then google only knows your links to your correspondents who use gmail, but not your links to those using hotmail, facebook, etc. (unless that info is revealed to google by putting multiple recipients in the to or cc entries in the message.)