RSS Box – RSS for websites that do not support RSS

Re:RSS Box – RSS for websites that do not support RSS by oefrha on Saturday June 20, 2020 @11:22AM (#23584425)

Feed aggregation services tend to minimize the frequency of requests especially for unpopular feeds since more requests = higher load for them as well. The incentives on the publishing and consuming sides align. Many services only offer to increase crawling frequencies for premium users, and even then only for a limited number of feeds. Not to mention they only need to crawl once for however many subscribers.

It’s really people who don’t use aggregation services and set their clients to update very frequently (say every minute) that pose a problem.

[ Parent | on HN ]

Re:RSS Box – RSS for websites that do not support RSS by hinkley on Saturday June 20, 2020 @12:15PM (#23584716)

An aggregator most likely would prioritize feeds by popularity and then run most of their crawlers 24/7. Telling some users that a popular feed has new entries and not telling the free tier sounds complicated. Unless the free tier is getting a read replica of the database, in which case it’s throttled only by how far behind the read replicas get.

Prioritizing lower subscribership feeds if a smaller number of premium customers add them makes some sense though.

[ Parent | on HN ]

Re:RSS Box – RSS for websites that do not support RSS by oefrha on Saturday June 20, 2020 @12:34PM (#23584812)

Of course popular feeds are crawled constantly, and no, there’s no “telling some users that a popular feed has new entries and not telling the free tier”. But you bet services don’t want to crawl those one-subscriber feeds (e.g. some websites have personal feeds for paid subscribers) constantly for free. And RSS Box feeds apparently tend to fall in the latter category.

[ Parent | on HN ]