olviko

Maybe. Don't think any of this existed when I need it...

1. The code is tiny with 90% of it dealing with RSS parsing and filtering. Using RX.NET wouldn't really simplify anything.

2. I wanted a library that I can integrate into my apps and run locally to avoid throttling, robots.txt and other BS Yahoo Pipes was suffering from.

I am also not a huge fan of RX... to put it mildly

Can’t really remember, sorry, that code was built for processing RSS feeds and data size was never an issue. I will take a look when get some free time…

But, I see what you mean. I had to deal with similar issues in commercial projects and the "pull" model (generators in Python ~ "yield return" in C#) almost never a good idea, especially when you have to have concurrent consumers. While callbacks are hard to combine, in C# it can be nicely abstracted with “async/await”, not sure how it is handled in Python, I stopped using it around 2.5

Nice project. I wrote something similar in C# long time ago [1]. Mostly to monitor job feeds and craigslist [2] :-) It supports RSS and Atom, async, various filters, deduplication, etc

Yahoo Pipes was a nice project, but as its popularity grew, it started getting blocked more and more. It was also hard to build and maintain pipelines with more than a few steps.

[1] https://github.com/olviko/RssPercolator

[2] https://github.com/olviko/RssPercolator/blob/master/RssPerco...

Just curious, why not Redis?

Same here. InfluxDb has a pretty nice DSL. I wish they switched their backend to something usable and mature instead of re-inventing the bicycle. Operationally, it is expensive to support specialized databases like Influx, unless it is your core business, I guess...

What are the benefits of introducing specialized time-series databases vs using Redis, Casandra, or some Sql database?

Checkout https://github.com/olviko/RssPercolator

Pipes would often get blocked by 3rd party data providers, which motivated me to roll my own library.

No graphical interface. Personally, I found Y! Pipes UI hard to work with when I needed to build large pipelines.

Checkout https://github.com/olviko/RssPercolator.

RssPercolator is a .NET library used for downloading, aggregating, and filtering RSS feeds. Developed out of frustration with Yahoo Pipes.

- RSS and Atom feed formats - Asynchronous multi-feed download - Multiple sources and multiple destinations - Feed filters (string match, wildcards, and Regex)

If you like Yahoo Pipes as a concept this library is for you ;-)

  - RSS and Atom feed formats
  - Asynchronous multi-feed download
  - Multiple sources and multiple destinations
  - Feed filters (string match, wildcards, and Regex