Upton: A Web Scraping Framework

Re:Upton: A Web Scraping Framework by hobonumber1 on Monday July 22, 2013 @04:36PM (#6086307)

I use YQL to do web scraping. It lets me do something like this:

`select * from data.html.cssselect where url="www.yahoo.com" and css="#news a"`

Could you elaborate on the benefits of using Upton instead of this?

[ Parent | on HN ]

Re:Upton: A Web Scraping Framework by joshaidan on Monday July 22, 2013 @04:44PM (#6086349)

I've always used this method:

http://railscasts.com/episodes/190-screen-scraping-with-noko...

As well as Mechanize when working with sites that require session cookies and all that.

I'm wondering too what the advantages of Upton are?

[ Parent | on HN ]

Re:Upton: A Web Scraping Framework by jeremybmerrill on Monday July 22, 2013 @04:52PM (#6086389)

AFAICT, YQL can only handle scraping individual pages that way.

Upton can scrape a whole set of pages. If you have a page that lists the pages you're interested in; suppose you're interested in HN commenters on front page posts, you could specify the front page URL and a selector for links to comment pages, and Upton would automatically scrape those pages and return them to you.

Upton could even write the commenter names to a CSV for you with just a filename and a CSS selector/XPath expression.

It's not stuff you couldn't do with YQL or Python/BeautifulSoup. But it's stuff that I didn't want to have to write over and over each time I wrote a new scraper.

[ Parent | on HN ]

Re:Upton: A Web Scraping Framework by hobonumber1 on Monday July 22, 2013 @09:23PM (#6087615)

Makes sense! Thanks for clarifying that.

[ Parent | on HN ]