Re:Scrape like the big boys
by ransom1538 on Friday November 05, 2021 @01:14PM
(#29122101)
I love the honey pot approach. Put tons of valued hrefs on the page that are invisible (css) that the scrapper would find. Then just rate limit that ip address and randomize the data coming back. Profit.
[ Parent
| on HN ]
Re:Scrape like the big boys
by histriosum on Saturday November 06, 2021 @01:03PM
(#29131859)
I think this falls into the "arms race" trap, though. If you can make an href invisible via CSS, then the scraper can certainly be written to understand CSS, and thus filter out the invisible hrefs..