News for hackers. Stuff that matters.

A comment on Using Scrapy to Build Your Own Dataset. Parent comment

Re:Using Scrapy to Build Your Own Dataset by eccfcco15 on Tuesday September 26, 2017 @10:58AM (#15339203)
Why? Isn’t it better to use a vps or two (or even just a vpn), and strong rate limiting? I.e. try not to be noticed regardless of the robots.txt.
[ Parent | on HN ]
Re:Using Scrapy to Build Your Own Dataset by JosephRedfern on Tuesday September 26, 2017 @11:28AM (#15339486)
Because it's considered polite.
[ Parent | on HN ]
Re:Using Scrapy to Build Your Own Dataset by slaymaker1907 on Tuesday September 26, 2017 @02:11PM (#15341173)
A lot of robots.txt exist to help crawlers automatically know which URLs are safe and which ones are mutable (i.e. the sign up link).

You should only ignore the robots.txt if you are very careful about what you are doing and have a very good reason for doing so.

[ Parent | on HN ]