Scrape like the big boys

Re:Scrape like the big boys by _jal on Friday November 05, 2021 @10:39AM (#29119925)

In a past life, we were consulting with a startup that offered a subscription data service. They were very sensitive about scrapers, especially on the time limited try-before-you-buy accounts, which competitors were abusing.

At their request, we built a method to flag accounts for data poisoning. Once flagged, those accounts would start getting plausible-ish looking garbage data.

It was pretty effective. One competitor went offline for a few days about a week after that started, and had a more limited offering when they came back up.

[ Parent | on HN ]

Re:Scrape like the big boys by scarygliders on Friday November 05, 2021 @10:48AM (#29120057)

That's a good way of going about dealing with this kind of abuse indeed. Wish I'd thought of doing that at the time, but due to the nature of this shop you didn't need a user account to browse the products/prices.

I'm now making an entirely new shop for them - I shall bear this in mind. Thanks for that!

[ Parent | on HN ]

Re:Scrape like the big boys by brightball on Friday November 05, 2021 @03:46PM (#29123953)

Yea. Detect them and mess with them is the only approach that seems to work for a lot of abusive activity. Banning doesn’t work because they will just start over from scratch. The only thing you can really do is make them think you haven’t “caught” them yet and during that stretch make sure their time is wasted.

[ Parent | on HN ]