Re:“Magic links” can end up in Bing search results, renderin...
by snowwrestler on Tuesday June 28, 2022 @10:46AM
(#31908364)
You are correct: a bot that is well-behaved (follows robots.txt directions) will not crawl your site if your robots.txt forbids crawling.
[ Parent
| on HN ]
There are no comments on this comment yet.