News for hackers. Stuff that matters.
Re:“Magic links” can end up in Bing search results, renderin... by gunapologist99 on Monday June 27, 2022 @09:01AM (#31893972)
> Warning: Don't use a robots.txt file as a means to hide your web pages from Google search results.

I realize that you are just the messenger and not the progenitor of that policy, so not addressing this to you, but: that is ridiculous. robots.txt is basically useless.

Re:“Magic links” can end up in Bing search results, renderin... by jeroenhd on Monday June 27, 2022 @12:18PM (#31896734)
Robots.txt is a mechanism for providing instructions to automated crawlers but I don't think they've ever been promised to be used when a URL is manually or automatically submitted through other means (i.e. another site linking to yours). In those cases, a single page will probably be crawled, but the rest of the domain probably won't.