If you have any https links on your site, those could be replaced by http links using something like sslstrip. Your site is now the weak link in the chain google->your site->linked site.

Of course sites should be using HSTS to help prevent this, but if the user is visiting the linked site for the first time, HSTS can't protect them.