News for hackers. Stuff that matters.

A comment on News.YC in Polar Coordinates. Parent comment

Re:News.YC in Polar Coordinates by bootload on Wednesday September 19, 2007 @03:26AM (#56771)
Nice hack nostro. What process did you use to scrape? Been working on a newsyc scraper

- extract & parse with python + beautifulSoup, display with YUI

- parsing the RSS feed for user, story points, story title and date posted

- then grabbing user details via homepage for karma, inception date

- exporting to xml

- skinning with YUI

Will display when ready, heres a screenshot ~ http://flickr.com/photos/bootload/1400863977/

[ Parent | on HN ]
Re:News.YC in Polar Coordinates by nostrademons on Wednesday September 19, 2007 @09:40AM (#56854)
Regexps. John tried a beautifulSoup parser, but it turned out to be more trouble than it was worth. Insert joke about "now you have two problems", but it works in a pinch, and it really only had to last till we showed off our projects at 8:00 PM ;-).

I don't mind sharing the code if PG doesn't mind the potential flood of screen scrapers. It just handles the front page and comment page for individual articles, and it's pretty compact - only 60 lines including doc comments.

[ Parent | on HN ]
Re:News.YC in Polar Coordinates by bootload on Thursday September 20, 2007 @12:50AM (#57255)
"... Regexps. John tried a beautifulSoup parser, but it turned out to be more trouble than it was worth ...'

Took me a bit of mucking around to get it working. My trick was just utilising the FOX HTML Validator to find the page structure, then CUT+PASTE the text into IDLE and call up BS in IDLE, then write the BS expression to oarse the string till I had the right data.

"... I don't mind sharing the code if PG doesn't mind the potential flood of screen scrapers. ..."

A better suggestion might be supply the raw data you collected as a service and let others do what they want with it. Most just want the data and are not particularly interested in the HOW. If they are interested in the HOW then it's better to point them to some scrapers to play with.

[ Parent | on HN ]