This is a completely fair question and is something anyone working in performance optimization (marketing, product, etc.) should be asking themselves all the time when looking at data and finding ways to suss it out.

In the case of retargeting ads, you can run tests to figure it out. When I was doing more on the marketing side, I used to find a non-profit and then show ads for the non-profit to my retargeting audiences. That way we could test to see what kind of value our retargeting ads had vs. a completely irrelevant retargeted ad.

For the most part, the relevant retargeted ad outperformed irrelevant by enough magnitude to make the retargeting campaign profitable.