Re:A steep rise of Hacker News in Google rankings
by factormeta on Sunday February 18, 2024 @07:34PM
(#39425023)
HN can always change their robot.txt and prevent google from crawling. Many of copy left friends do that to their blog and site to prevent google/bing from indexing them.
[ Parent
| on HN ]
Re:A steep rise of Hacker News in Google rankings
by sitkack on Monday February 19, 2024 @06:30AM
(#39428630)
This is really counter productive towards furthering the cause.
Re:A steep rise of Hacker News in Google rankings
by xp84 on Tuesday February 20, 2024 @11:05PM
(#39450018)
Just so you know, that doesn’t stop Google from indexing you anymore. It only stops Googlebot from crawling. In other words it’s strictly a place to deny permission or set limits on a technical level. If you do this they will still have what they already crawled and if there are links to it on the Internet they will keep it in the index.
If you don’t want something to be in Google on a policy level, you now have to add the noindex meta tag on every page.