News for hackers. Stuff that matters.

A comment on QVC Sues Shopping App for Web Scraping That Allegedly Triggered Site Outage. Parent comment

Re:QVC Sues Shopping App for Web Scraping That Allegedly Tri... by acdha on Friday December 05, 2014 @02:00PM (#8706194)
FWIW, every time I've seen what looked like a major search engine ignoring rate-limits (either Crawl-Delay or webmaster tools settings) a check of the actual IPs being used showed that it was someone spoofing a well-known User-Agent, which left you needing some other form of rate-limiting either way.
[ Parent | on HN ]
Re:QVC Sues Shopping App for Web Scraping That Allegedly Tri... by johngd on Friday December 05, 2014 @02:38PM (#8706413)
Very true, however the incidents I reference this was definitely not the case.

In fact for a while we would get Bing (MSN bot back then) crawl us everyday at the same time, almost on the dot.

Let me plug project honeypot (which I am in no way affiliated with). This is truly an awesome, and surprisingly accurate, free, service that does an amazing job at collecting heuristics on suspicious IP activity and exposing it in a easy to interpret way..

http://www.projecthoneypot.org/index.php

[ Parent | on HN ]
Re:QVC Sues Shopping App for Web Scraping That Allegedly Tri... by acdha on Friday December 05, 2014 @03:47PM (#8706799)
Project Honeypot is indeed great – I've been running their collectors for years on a few spare domains, racking up a fair number of harvesters.
[ Parent | on HN ]