There are two different ways ads can be displayed. Either the message is scanned for a topic and ads are displayed to you based on it, or they build a profile about you, like you described. You can opt out of that:

https://www.google.com/settings/u/0/ads/preferences/?hl=en