News for hackers. Stuff that matters.
Re:The Web is missing an essential part of infrastructure: a... by amitamb on Sunday April 21, 2019 @08:15PM (#19715402)
Apart from that Common Crawl respects robots.txt (which makes sense) so many sites you expect to see there are not indexed. Netflix, Facebook LinkedIn and many more. If common-crawl sees serious adoption those sites will modify their robots.txt but it's and chicken/egg problem.
Re:The Web is missing an essential part of infrastructure: a... by amelius on Monday April 22, 2019 @10:03AM (#19718987)
There is a simple solution: if companies do not respect do-not-track then why should we respect robots.txt?
Re:The Web is missing an essential part of infrastructure: a... by nostrademons on Tuesday April 23, 2019 @01:14AM (#19725769)
Because then you end up in an arms race that the little guy usually does not win.

There are a significant number of crawlers out there that don't respect robots.txt. The usual response to them isn't to roll over dead, it's to get CloudFlare (on the technological end) and/or sic the lawyers on them (for CFAA, IP, or ToS violations).