News for hackers. Stuff that matters.
Re:Why can’t a bot tick the 'I'm not a robot' box? by learc83 on Wednesday February 13, 2019 @05:01PM (#19157522)
At work we have a scraper that likes to use a particularly expensive search query using hundreds of different ips dozens of times per second, all so that they can scrape data that is freely available from us as an XML feed at a different url.

Every time I found a way to fingerprint and block them, they'd change their bot to avoid detection. Captcha for rate limiting seemed like the least bad option.

However, eventually I decided that instead of blocking them, I'd return random results from the db. They weren't checking the data too thoroughly because to this day they haven't changed the bot to avoid my most recent pattern detection. They're still merrily scraping useless data.

Re:Why can’t a bot tick the 'I'm not a robot' box? by herpderp3dtwerp on Wednesday February 13, 2019 @08:24PM (#19158876)
Awesome idea giving them crappy data. I do the same thing with loyalty programs. Someone has already registered XXX-555-1234 as a phone number in your area code. I like to think I'm adding a little bit of chaos to their data tracking.
Re:Why can’t a bot tick the 'I'm not a robot' box? by seotut2 on Wednesday February 13, 2019 @06:59PM (#19158383)
Rate limiting can be achieved without actually inconveniencing regular users. Put a exponentially growing delay on the server's response for requests coming in too quick succession.
Re:Why can’t a bot tick the 'I'm not a robot' box? by learc83 on Wednesday February 13, 2019 @09:26PM (#19159161)
The problem was that with the sheer number of ip addresses they were using, and with the rate that some normal users used that particular endpoint, regular users would have been inconvenienced--either by being forced to wait or by being forced to do a captcha.

I would have set it to only show the captcha if the delay was active, so users would effectively have had a choice of wait or captcha.

Re:Why can’t a bot tick the 'I'm not a robot' box? by robk on Thursday February 14, 2019 @05:55AM (#19160990)
Why not add a "PLEASE USE THE XML DUMMY!" line to the returned results when you figure out it's them?
Re:Why can’t a bot tick the 'I'm not a robot' box? by learc83 on Thursday February 14, 2019 @05:30PM (#19166699)
Tried that. They didn't seem to care.