News for hackers. Stuff that matters.

A comment on Avoiding bot detection: How to scrape the web without getting blocked?.

Re:Avoiding bot detection: How to scrape the web without get... by adinosaur123 on Sunday October 31, 2021 @06:25PM (#29061067)
Are there any court cases that provide precedence regarding the legality of web scraping?

I'm currently looking for ways to get real estate listings in a particular area and apparently the only real solution is the scrape the few big online listing sites.

[ Parent | on HN ]
Re:Avoiding bot detection: How to scrape the web without get... by Grimm1 on Sunday October 31, 2021 @06:35PM (#29061139)
https://en.m.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn

That’s one of the bigger ones. Unfortunately recent events means scraping is still a gray area.

[ Parent | on HN ]
Re:Avoiding bot detection: How to scrape the web without get... by amelius on Sunday October 31, 2021 @06:51PM (#29061245)
Legal gray areas are perfect for growth hacking. Just look at Uber and AirBnb.
[ Parent | on HN ]
Re:Avoiding bot detection: How to scrape the web without get... by Grimm1 on Sunday October 31, 2021 @08:14PM (#29061778)
And even bigger growth hack for a lot of companies would be having scraping protected by law.
[ Parent | on HN ]
Re:Avoiding bot detection: How to scrape the web without get... by omgwtfbyobbq on Sunday October 31, 2021 @06:54PM (#29061268)
Do you mean this case?

https://en.m.wikipedia.org/wiki/Van_Buren_v._United_States

I think it only applies to systems that aren't available to the general public, which in this case was the GCIC. Anything that is available to the public, even if it requires some sort of registration, would I think be legal to scrape. YMMV though.

[ Parent | on HN ]
Re:Avoiding bot detection: How to scrape the web without get... by adanto6840 on Sunday October 31, 2021 @06:56PM (#29061281)
I was involved in a scraping-related case, though in my situation we were scraping public domain data/facts/public domain media. Email me if you'd like additional info. :)

More related to the submission content -- at the time we used rotating proxies, both in-house & external (ProxyMesh - still exists & only good things to say about it); they allowed us to "pin" multiple requests to an IP or to fetch a new IP, etc...

[ Parent | on HN ]