News for hackers. Stuff that matters.
Re:“Magic links” can end up in Bing search results, renderin... by rsbadger on Monday June 27, 2022 @07:27AM (#31893066)
It was blocked by robots.txt but Bing chose to ignore it. I even tried "blocking" the URLs in Bing webmaster tools today and this was the response:

"Block request denied We found that the URL submitted for block is important for Bing users and hence cannot be blocked through Bing Webmaster Tools.

We recommend that the best way to block URLs in this scenario is to add NOINDEX meta-tag to the HTML header of the page."

Re:“Magic links” can end up in Bing search results, renderin... by cube00 on Monday June 27, 2022 @08:28AM (#31893584)
That is baffling logic. Sure, they think they know best and want to ignore the wishes of the owner of the web site. Why then respect a NOINDEX meta-tag instead of robots.txt?
Re:“Magic links” can end up in Bing search results, renderin... by rsbadger on Monday June 27, 2022 @08:41AM (#31893733)
Exactly - seems the safest way is to explicitly block known bots by user agent from even reaching pages you don't want indexed.