I've even bought a few things I found through Facebook or Instagram ads, things that I wasn't aware of and that I enjoy, timeular and old school safety razors for example, proving that advertising can be a win-win game.
No, what they do is buy point-of-sale data from stores, often through companies like Nielsen and larger. NOW they know what you're buying, and Banana Republic ain't gonna stop telling them, because the market research aggregators pay them back by telling them who is buying their stuff, like people whose credit card usage is 45% directed at clothing stores.
It's far more likely that the Gap is uploading your itemized receipt data to advertising networks (complete w/ that email address they always ask you for and your CC info) under the guise of improving their own retargeting.
Some time later I was in my closet and still pondering how Facebook could possibly be so wrong as to think I was interested in pants of all things. On a whim, I decided to count my pants. And you know what? I had a lot of pants. Triple digits. Way more pants than I realized I had and way more pants than any one person could possibly need. It turns out, I _was_ pretty interested in pants.
Facebook, without ever having seen my closet, had a better understanding of my wardrobe than I did.