News for hackers. Stuff that matters.

A comment on Overview of Text Extraction Algorithms. Parent comment

Re:Overview of Text Extraction Algorithms by tha-dude on Sunday March 20, 2011 @08:41AM (#2346128)
I've been dabbling in content-scraping, what bugs me is that with all the AJAX trickery that's going on, merely analyzing the XHTML source doesn't get you very far in many cases. Executing the page (JS, DOM and all) via browser-programming is an option but of course quite expensive. A headless browser is what's needed!
[ Parent | on HN ]
Re:Overview of Text Extraction Algorithms by pkandathil on Monday March 21, 2011 @12:49PM (#2350498)
Yeah. I think that is the challenge. A good way to get around the AJAX problem is to see if a site has an RSS feed and use that to extract content. I wish sites had a url for bots built in so you didnt have to do all this fancy stuff to extract the content.
[ Parent | on HN ]
Re:Overview of Text Extraction Algorithms by buss on Monday March 21, 2011 @10:12PM (#2352677)
Many of the big sites will feed you non-ajax content if you're the googlebot.
[ Parent | on HN ]