Re:Detecting Chrome headless, the game goes on
by klenwell on Friday July 19, 2019 @01:11PM
(#20480510)
I would assume that any site that was implementing JS-level blocks also has the appropriate robots.txt file in place.
[ Parent
| on HN ]
Re:Detecting Chrome headless, the game goes on
by greglindahl on Friday July 19, 2019 @04:17PM
(#20482085)
That's not true in the actual web, however.
The best example is a large number of unimportant sites that send 429 errors for /robots.txt if they think it's a scraper. A 4xx result for robots.txt is considered to mean no robots.txt for most crawlers. So the website is getting the reverse of what it thought it was getting.