Re:Detecting Chrome headless
by stordoff on Saturday August 05, 2017 @08:02PM
(#14938790)
Could you have the content of the page explain that its a bot trap (so screen reader users know its not worth visiting) and only block if, say, an IP visits it multiple times in a short window?
Wouldn't block specialised scrapers as they would know to avoid that URL/link (though they'd probably work it out anyway), but would still limit more broad crawlers.
Wouldn't block specialised scrapers as they would know to avoid that URL/link (though they'd probably work it out anyway), but would still limit more broad crawlers.