News for hackers. Stuff that matters.
Re:Full Text RSS Feed: Get the whole feed and nothing but th... by grayrest on Tuesday March 01, 2011 @06:48PM (#2277807)
What's he using to pull out the articles? I had a hacky version set up using the Readability algorithm but never bothered to make it public.
Re:Full Text RSS Feed: Get the whole feed and nothing but th... by yesimahuman on Tuesday March 01, 2011 @07:55PM (#2278011)
Boilerpipe is by far the best tool for this that I've ever found (http://code.google.com/p/boilerpipe/). I'd be interested to hear if he is using something better, but I'd be surprised if he is.

I think this is a great idea and very similar to a lot of stuff I have worked on recently. It's cool to see so much interest in these text-related services.

Re:Full Text RSS Feed: Get the whole feed and nothing but th... by nikcub on Wednesday March 02, 2011 @02:11AM (#2278740)
Thanks for that link - exactly what I was looking for

btw I know that at Techmeme, Gabe spent years perfecting his story parsing for the 50k+ sites he tracks. Even something that would seem simple such as parsing the date of a story from a webpage has a ridiculous number of permutations that you have to grep for.

Re:Full Text RSS Feed: Get the whole feed and nothing but th... by benmccann on Tuesday March 01, 2011 @11:09PM (#2278407)
I don't think it's quite as good as what he's doing though. He has the title and date specifically pulled out and he doesn't have any extra text included. I think he manually handles CNN. If I try a HuffPost feed it doesn't work at all.
Re:Full Text RSS Feed: Get the whole feed and nothing but th... by yesimahuman on Wednesday March 02, 2011 @12:47AM (#2278600)
Yea, I'd be curious to see exactly what he's doing. I can only guess there is a heuristic which results in a lot of failed feed processing noticed on here (I know it's just a weekend project :)) that doesn't generalize well. Boilerpipe, in my experience, works very well on almost all news/blog type content. Finding the date in the first few sentences and the title are extra heuristics that can be added later.

EDIT: The date and title are in the RSS feed already! No further analysis needed.

Re:Full Text RSS Feed: Get the whole feed and nothing but th... by osmandros on Wednesday March 02, 2011 @03:56AM (#2278900)
That library is very robust. On the company I was working last year, We built a news crawler using that tool, and adapted it as a plugin for nutch.
Re:Full Text RSS Feed: Get the whole feed and nothing but th... by gregorym on Wednesday March 02, 2011 @03:45AM (#2278883)
That is freaking awesome! http://boilerpipe-web.appspot.com/
Re:Full Text RSS Feed: Get the whole feed and nothing but th... by beagledude on Tuesday March 01, 2011 @10:52PM (#2278382)
Goose article extractor has a full suite of unit tests and also does pure text and image extractions: https://github.com/jiminoc/goose
Re:Full Text RSS Feed: Get the whole feed and nothing but th... by k1m on Thursday March 03, 2011 @04:42PM (#2285901)
Possible to do it with Readability - my PHP port is here: http://www.keyvan.net/2010/08/php-readability/ - and similar tool (free software, can be self-hosted) using the PHP Readability here: http://fivefilters.org/content-only/