Yeah, running this thing publicly would be a huge mess from a copyright perspective, since it literally re-hosts everything as a core part of how it works.

As it is, I think I'm OK, since it's basically just a "website DVR" type thing, for my own use.

Really, if nothing else, the project has been enormously educational for me. I've learnt a boatload about distributed systems, learned a bit of SQL, dicked about with databases a bunch, and actually experienced deploying a complex multi-component application across multiple disparate data centers.

This project is really cool. Last year I was looking into open source projects that implement something like Readability so that I could scrape articles from my RSS feeds and turn them into plaintext. But I didn't find anything that blew me away. The best I got was stealing the implementation from Firefox, and I lost interest before I could make it worthwhile. (Now revisiting the idea, I wonder why I never thought of passing a user-agent from a mobile browser... Probably would have helped a lot.)

I see you don't have a license listed on GitHub. Do you have a license in mind for these?

It's probably GPL, I'll have to figure out my dependencies and see what it's infected with. I tend to err BSD on my own cruft.

This isn't quite as fancy as readability, though I integrated a port of readability for a while. Now I just write a ruleset for a site that has stuff that interests me.

Note: I stuck it under BSD license.

Similar, paid solution: https://scrapinghub.com/crawlera/