Re:“Magic links” can end up in Bing search results, renderin...
by jeroenhd on Monday June 27, 2022 @12:18PM
(#31896734)
Robots.txt is a mechanism for providing instructions to automated crawlers but I don't think they've ever been promised to be used when a URL is manually or automatically submitted through other means (i.e. another site linking to yours). In those cases, a single page will probably be crawled, but the rest of the domain probably won't.
[ Parent
| on HN ]
There are no comments on this comment yet.