News for hackers. Stuff that matters.
Re:Avoiding bot detection: How to scrape the web without get... by jhgb on Sunday October 31, 2021 @09:29PM (#29062182)
https://developer.mozilla.org/en-US/docs/Web/HTTP/Status#cli...

> Although the HTTP standard specifies "unauthorized", semantically this response means "unauthenticated". That is, the client must authenticate itself to get the requested response.

So it would seem that it actually doesn't positively imply that you're NOT authorized.

Which kind of makes sense; machines can't detect legality of things, just that certain procedural niceties haven't been observed.

Re:Avoiding bot detection: How to scrape the web without get... by marginalia_nu on Sunday October 31, 2021 @09:41PM (#29062252)
Fine, send a 403 then.

> The client does not have access rights to the content; that is, it is unauthorized, so the server is refusing to give the requested resource.

Machines don't have any legal responsibility, bot-operators do. Which is why respecting these things is sort of important. At any rate, 40x does not mean "try again with a different user agent and another IP"

Re:Avoiding bot detection: How to scrape the web without get... by hattmall on Monday November 01, 2021 @12:33AM (#29063019)
403 is per request, not requester. I get random 403s when just browsing some websites. Does that mean I should close the browser and not hit refresh for fear of breaking some wire fraud unauthorized access law?
Re:Avoiding bot detection: How to scrape the web without get... by marginalia_nu on Monday November 01, 2021 @04:55AM (#29064351)
If you go by the semantics of what the 403 code means, absolutely, that's excatly what the status code means.

In practice there's of course nuance, like anyone will occasionally type in the wrong password on a log-in screen, maybe try again and then realize it was the wrong log-in prompt. That's mostly fine.

That's different from deliberate trying to circumvent a measure like this. If you are doing the stuff in the link, you are absolutely crossing a line and you know it.

There's a large difference between "I got a 403 so I hit F5 once" and "I got a 403 so I used a residential proxy and spoofed my user-agent".

Re:Avoiding bot detection: How to scrape the web without get... by jhgb on Monday November 01, 2021 @08:23AM (#29065793)
I thought the semantics of 403 was "look for the actual semantics in the response body".