Re:Ask YC: Grabbing content
by spage on Thursday April 17, 2008 @08:54AM
(#166263)
My favorite technique is:
wget URL > HTML
tidy HTML > XHTML
xslt [identity transform based content extraction] XHTML > XML
XML > DB
The whole process glued together with PERL or shell scripts. Depending on how you construct your content extraction, this technique can weather lots of the inevitable content style changes and easily adjusts when changes need to be made.
wget URL > HTML tidy HTML > XHTML xslt [identity transform based content extraction] XHTML > XML XML > DB
The whole process glued together with PERL or shell scripts. Depending on how you construct your content extraction, this technique can weather lots of the inevitable content style changes and easily adjusts when changes need to be made.