Re:“Magic links” can end up in Bing search results, renderin...
by gunapologist99 on Monday June 27, 2022 @09:01AM
(#31893972)
> Warning: Don't use a robots.txt file as a means to hide your web pages from Google search results.
I realize that you are just the messenger and not the progenitor of that policy, so not addressing this to you, but: that is ridiculous. robots.txt is basically useless.
Re:“Magic links” can end up in Bing search results, renderin...
by jeroenhd on Monday June 27, 2022 @12:18PM
(#31896734)
Robots.txt is a mechanism for providing instructions to automated crawlers but I don't think they've ever been promised to be used when a URL is manually or automatically submitted through other means (i.e. another site linking to yours). In those cases, a single page will probably be crawled, but the rest of the domain probably won't.
Re:“Magic links” can end up in Bing search results, renderin...
by rsbadger on Monday June 27, 2022 @07:51AM
(#31893234)
They are now - I didn't think I had to as all the pages are naturally behind a login, it never crossed my mind that Bing would follow email links, let alone index them in search results.
> Warning: Don't use a robots.txt file as a means to hide your web pages from Google search results. If other pages point to your page with descriptive text, Google could still index the URL without visiting the page. If you want to block your page from search results, use another method such as password protection or noindex.
From https://developers.google.com/search/docs/advanced/robots/in...