News for hackers. Stuff that matters.
Re:Ask HN: Best practices for ethical web scraping? by the8472 on Saturday April 04, 2020 @02:52PM (#22780533)
The other way around may be easier, i.e. excluding known datacenter ranges. There are some commercial databases for that, i'm not sure if there are any free ones. But you can also do this manually by running a whois on an IP and then extracting the ranges from the whois response and caching then. Then you can look at the orgname or something like that. You can also download the whois databases from the RIRs, but they don't contain the information what kind of entities they are.

    $ dig +short reddit.com
    151.101.1.140


    $ whois 151.101.1.140

    NetRange:       151.101.0.0 - 151.101.255.255
    CIDR:           151.101.0.0/16
    OrgName:        Fastly
    [...]
So if you see a known hoster here then you can exclude it from your statistics.

There are no comments on this comment yet.