Re:Go Read: One Year with Money and App Engine
by mjibson on Wednesday May 21, 2014 @01:48PM
(#7779634)
goread has pubsubhubbub support. penny arcade doesn't use it, though. The penny arcade feeds are indeed giving errors. I'll look into those.
[ Parent
| on HN ]
Re:Go Read: One Year with Money and App Engine
by ecesena on Wednesday May 21, 2014 @09:16PM
(#7781635)
At Theneeds we use a "sliding windows" approach to deal with polling. Say you run the scraper every hour. Each feed F_i is scraped once every n_i polls. If the feed returns more than his average news, then n_i gets decreased, while if the feed returns no news, it gets increased.
Perhaps with a similar trick you can run your scraper more frequently on some feeds, still keeping the cost under control.
Thanks! I don't really know when the website has a problem, when goread has a problem, and when I have a problem, so I end up assuming that it's goread problem. I like how the desktop web version gives a "last refreshed" and "next refreshed" indication.