Web Scraping in 2016

Re:Web Scraping in 2016 by nickysielicki on Tuesday August 23, 2016 @08:41PM (#12348900)

This project is really cool. Last year I was looking into open source projects that implement something like Readability so that I could scrape articles from my RSS feeds and turn them into plaintext. But I didn't find anything that blew me away. The best I got was stealing the implementation from Firefox, and I lost interest before I could make it worthwhile. (Now revisiting the idea, I wonder why I never thought of passing a user-agent from a mobile browser... Probably would have helped a lot.)

I see you don't have a license listed on GitHub. Do you have a license in mind for these?

[ Parent | on HN ]

Re:Web Scraping in 2016 by fake-name on Tuesday August 23, 2016 @09:04PM (#12348961)

It's probably GPL, I'll have to figure out my dependencies and see what it's infected with. I tend to err BSD on my own cruft.

This isn't quite as fancy as readability, though I integrated a port of readability for a while. Now I just write a ruleset for a site that has stuff that interests me.

[ Parent | on HN ]

Re:Web Scraping in 2016 by fake-name on Thursday August 25, 2016 @01:33PM (#12360846)

Note: I stuck it under BSD license.

[ Parent | on HN ]