News for hackers. Stuff that matters.

A comment on Only Google is really allowed to crawl the web.

Re:Only Google is really allowed to crawl the web by p-sharma on Friday March 26, 2021 @01:03PM (#26594413)
Maybe a naïve question but what prevents Knuckleheads’ from ignoring the robots.txt and crawl the side anyway? And if it's so easy to do, how does Google have a monopoly on crawling then?
[ Parent | on HN ]
Re:Only Google is really allowed to crawl the web by foobar33333 on Friday March 26, 2021 @05:38PM (#26597274)
On smaller sites, nothing usually. But on bigger sites you will be blocked. You will probably be blocked even if you do follow robots.txt
[ Parent | on HN ]
Re:Only Google is really allowed to crawl the web by judge2020 on Friday March 26, 2021 @01:59PM (#26595013)
It's just rude to do so, and there are some technical issues with doing that as well (such as crawling admin panel which might trigger backend alarms/security alerts). Google also doesn't have a legal monopoly on crawling, only a natural monopoly thanks to a lot of websites independently choose to only allow Google and Bing because of the many issues with third-party crawlers (eg. crawling all pages at once, costing money/slowing down the site[0]).

0: https://news.ycombinator.com/item?id=26593722

[ Parent | on HN ]
Re:Only Google is really allowed to crawl the web by sn_master on Friday March 26, 2021 @11:00PM (#26599525)
Their IP range can be blocked. Google has known IP ranges and if the website admin wants can allow them based on that rather than robots.txt, and that would really mess up everyone else.
[ Parent | on HN ]