Re:Cleaning up bad bots
by ikeboy on Monday September 23, 2019 @05:06PM
(#21053656)
if they're running on AWS, which most crawlers are not
When I've run scraping software in the past I used DigitalOcean, which doesn't contain a requirement to abide by robots.txt. As far as I can tell it's both legal and consistent with their ToS to run a program that makes purchases on a website.
When I've run scraping software in the past I used DigitalOcean, which doesn't contain a requirement to abide by robots.txt. As far as I can tell it's both legal and consistent with their ToS to run a program that makes purchases on a website.