“Magic links” can end up in Bing search results, rendering them useless

Re:“Magic links” can end up in Bing search results, renderin... by boxed on Monday June 27, 2022 @04:57PM (#31900695)

I have a disallow all robots.txt for a production system. Have had from the beginning.

Bing indexes it. This is my first major security incident and I have no idea how to fix this without making everything totally shitty for the users.

[ Parent | on HN ]

Re:“Magic links” can end up in Bing search results, renderin... by efreak on Thursday June 30, 2022 @09:25AM (#31932069)

Some services ignore global disallow, but will respect rules explicitly targeted at them.

[ Parent | on HN ]

Re:“Magic links” can end up in Bing search results, renderin... by boxed on Thursday June 30, 2022 @03:07PM (#31937074)

I've put in a hard block for all crawlers on all pages. Works for my scenario I think. Hopefully they don't lie in their user agent. Then it's going to be really bad.

[ Parent | on HN ]