News for hackers. Stuff that matters.

A comment on Avoiding bot detection: How to scrape the web without getting blocked?. Parent comment

Re:Avoiding bot detection: How to scrape the web without get... by Spivak on Monday November 01, 2021 @11:03AM (#29067535)
But you have no freedom when contacting my servers. I can send you 403's just because I don't like your face. There is zero entitlement that you have any access to my servers in any way I don't permit. If I say no automated access then on what grounds to you have to do it anyway?

> Why is Google the only one who can do it?

Because site operators explicitly allow them automated access. If you want the same treatment you have to ask for it.

[ Parent | on HN ]
Re:Avoiding bot detection: How to scrape the web without get... by matheusmoreira on Monday November 01, 2021 @11:21AM (#29067854)
> I can send you 403's just because I don't like your face.

Sure. At least then you're being honest. If you hate me, it doesn't matter what user agent I use to access your site. Browsers, scripts, they are all me.

[ Parent | on HN ]
Re:Avoiding bot detection: How to scrape the web without get... by Spivak on Monday November 01, 2021 @12:15PM (#29068794)
Why do you think it’s dishonest to send 403s to bots but not browsers? Method of access matters — you the human might have access to your safety deposit box but the bank is still allowed to make rules about your access — like you have to come during business hours and you can’t send someone on your behalf.
[ Parent | on HN ]
Re:Avoiding bot detection: How to scrape the web without get... by matheusmoreira on Tuesday November 02, 2021 @02:11PM (#29085178)
> you can’t send someone on your behalf

I totally can though. If sign a document saying another person can do such and such on my behalf, that person can totally do that. Yes, even at the bank. No different from a user agent, really.

[ Parent | on HN ]