Avoiding bot detection: How to scrape the web without getting blocked?

Re:Avoiding bot detection: How to scrape the web without get... by arp242 on Sunday October 31, 2021 @11:07PM (#29062669)

> I really don't believe "The site owner decides what you can and cannot do." statement. What is the base of this? This does not seem to apply to anything in the real world.

It kind of does though; if I own a store and say that only people with hats can enter, then I'm free to do so. Silly? Yes. Legal? Also yes.

There are some circumstances where it's not legal, mostly centred around discrimination. Details on this differ per jurisdiction, but generally speaking you have a right to refuse customers.

[ Parent | on HN ]

Re:Avoiding bot detection: How to scrape the web without get... by geysersam on Monday November 01, 2021 @02:26AM (#29063600)

Interesting discussion.

To me it seems sending a http request is somewhere inbetween looking (legal) and entering (illegal if not permitted).

However, most important is that the web as a system makes positive interactions easy and negative difficult. We have already found some set of constraints achieving this for interactions in public city streets. But it's not obvious the same rules (that we have internalized) have the same effect in another medium of communication.

[ Parent | on HN ]