News for hackers. Stuff that matters.

A comment on Removing my site from Google search. Parent comment

Re:Removing my site from Google search by vadfa on Sunday October 03, 2021 @02:38PM (#28739469)
Or even better, iptables rules :P
[ Parent | on HN ]
Re:Removing my site from Google search by TedDoesntTalk on Sunday October 03, 2021 @02:45PM (#28739519)
Doesn’t that mean you have to know every IP Address used by Google bot now and in the future?
[ Parent | on HN ]
Re:Removing my site from Google search by judge2020 on Sunday October 03, 2021 @03:08PM (#28739699)
The way to check googlebot (in a way that will be resistant to expansion of Googlebot's IP ranges in the future) is to perform hostname lookup, with dns lookup as well to verify that the rDNS isn't a lie: https://developers.google.com/search/docs/advanced/crawling/...
[ Parent | on HN ]
Re:Removing my site from Google search by lucb1e on Sunday October 03, 2021 @03:14PM (#28739749)
Indeed, this was one of the things I considered (note I'm not OP), but then I didn't really want to rely on DNS. https://duckduckgo.com/?q=it's+always+DNS
[ Parent | on HN ]
Re:Removing my site from Google search by AshamedCaptain on Sunday October 03, 2021 @03:05PM (#28739668)
Not a very hard problem; after all, many websites allow full access to Googlebot IP ranges yet show a paywall to everyone else (including competing search engines).

I also happen to ban Google ranges on multiple less-public sites specially since they completely ignore robots.txt and crawl-delay.

[ Parent | on HN ]
Re:Removing my site from Google search by chihuahua on Sunday October 03, 2021 @08:50PM (#28741893)
Is that how archive.vn works? I've always wondered how they are able to get the full text of paywalled sites like Wall Street Journal who give 0 free articles per month.
[ Parent | on HN ]