Re:Show HN: robots.txt as a service, check web crawl rules t...
by fooock on Monday July 29, 2019 @01:12PM
(#20556888)
Yes, in the practice people sometimes don't want to be polite with webmasters, and choose not obey robots.txt rules. Thanks for the suggestion!
Re:Show HN: robots.txt as a service, check web crawl rules t...
by itsmefaz on Tuesday July 30, 2019 @01:42AM
(#20562371)
Exactly, your service could definitely be used as an alternate to parsing robots.txt (which traditionally is in xml) to a more standard json parsing. Along with the advantages that comes with making it REST.
I understand the unethical nature of the above method, however, I see it happening quite a lot in practice.