News for hackers. Stuff that matters.

A comment on Overview of Text Extraction Algorithms. Parent comment

Re:Overview of Text Extraction Algorithms by pkandathil on Monday March 21, 2011 @12:49PM (#2350498)
Yeah. I think that is the challenge. A good way to get around the AJAX problem is to see if a site has an RSS feed and use that to extract content. I wish sites had a url for bots built in so you didnt have to do all this fancy stuff to extract the content.
[ Parent | on HN ]
Re:Overview of Text Extraction Algorithms by buss on Monday March 21, 2011 @10:12PM (#2352677)
Many of the big sites will feed you non-ajax content if you're the googlebot.
[ Parent | on HN ]