Re:Ask HN: Best practices for ethical web scraping?
by capableweb on Saturday April 04, 2020 @04:04PM
(#22781059)
What I've done in the past is to pull down all the IPs of request I see, filter by unique, do whois for each one of them (you're gonna need to have a backoff/rate limit here as whois services are usually rate limited) and save the organization name, ASN and CIDR blocks, again filter by uniqueness, then create a new list with the organizations of interest and match with the CIDR blocks. Now you have an allow/blocklist you can use.
[ Parent
| on HN ]
There are no comments on this comment yet.