Ferret – Declarative web scraping

Re:Ferret – Declarative web scraping by hinkley on Tuesday October 02, 2018 @12:52PM (#18123404)

In late 2000 at the tail end of the bubble bursting, there was a search engine company trying to build on a platform of push notifications instead of crawling.

I know that at my current company the plurality if traffic comes from crawlers. We don’t want to throttle them because that’s biting the hand that feeds but it sure sucks.

I wonder often, how many crawlers you need before it’s cheaper for a website to volunteer up when pages change or new ones arrive.

[ Parent | on HN ]

Re:Ferret – Declarative web scraping by th0ma5 on Tuesday October 02, 2018 @01:26PM (#18123731)

Maybe Pointcast? https://en.wikipedia.org/wiki/PointCast_(dotcom)

[ Parent | on HN ]

Re:Ferret – Declarative web scraping by greglindahl on Tuesday October 02, 2018 @01:26PM (#18123746)

There are N existing standards for websites to notify crawlers about new content. Few websites use them, and the ones that do are often buggy. A few are even malicious.

I'm looking forward to the N+1th standard.

[ Parent | on HN ]