Re:Only Google is really allowed to crawl the web
by indymike on Friday March 26, 2021 @03:47PM
(#26596154)
Broadly speaking, robots.txt files are often ignored. I used to run a fairly large job ad scraping organization, and we would be hired by companies (700 of the fortune 1000 used us) to scrape the job ads from their career pages, and then post those jobs on job boards. 99 of 100 times, the robots file would disallow us to scrape. Since we were being paid by that company's HR team to scrape, we just ignored it because getting it fixed would take six months and 22 meetings.
[ Parent
| on HN ]
Re:Only Google is really allowed to crawl the web
by chmod775 on Friday March 26, 2021 @05:41PM
(#26597294)
> Broadly speaking, robots.txt files are often ignored.
If you wanna go nuclear on people who do that, include an invisible link in your html and forbid access to that URL in your robots.txt, then block every IP who accesses that URL for X amount of time.
Don't do this if you actually rely on search engine traffic though. Google may get pissed and send you lots of angry mail like "There's a problem with your site".
Re:Only Google is really allowed to crawl the web
by jedberg on Friday March 26, 2021 @06:33PM
(#26597819)
> Don't do this if you actually rely on search engine traffic though. Google may get pissed and send you lots of angry mail like "There's a problem with your site".
Ah, but of course you would exclude Google's published crawler IPs from this restriction, because that is exactly what they want you to do.