News for hackers. Stuff that matters.
Re:Dato.rss: The best RSS Search experience by marginalia_nu on Saturday June 18, 2022 @11:48PM (#31795998)
I've dabbled with using RSS feeds to inform the crawling of my search engine as well.

I think the big issue with leaning too heavily into RSS without much human intervention is that many CMS platforms generate an absurd number of RSS feeds, and it can be hard bordering on impossible to figure out which of them is canonical. Some (but not all) of these sites also tend to be fairly spammy low-value domains.

Re:Dato.rss: The best RSS Search experience by derekzhouzhen on Sunday June 19, 2022 @12:27AM (#31796167)
CMSs are not bad. Worse are the various forums. But the worst is nitter.net, which basically re-exposed the cesspool of twitter in RSS.
Re:Dato.rss: The best RSS Search experience by zaik on Sunday June 19, 2022 @04:44AM (#31797491)
Nitter RSS feeds are the only way I follow people on Twitter. I'm glad they exist.
Re:Dato.rss: The best RSS Search experience by derekzhouzhen on Sunday June 19, 2022 @07:36AM (#31798374)
I understand that nitter serve a purpose. However, for OP who is trying to crawl high quality content such as blogs and news automatically, nitter is a nightmare.
Re:Dato.rss: The best RSS Search experience by marginalia_nu on Sunday June 19, 2022 @07:49AM (#31798437)
Nitter is not that big of a deal, as they're just one website. Easy to block.