Re:How to write a crawler
by drakaal on Sunday September 29, 2013 @08:35PM
(#6467189)
For http://www.samuru.com we store our crawl in Google's DataStore. It gets big. By not storing all of the page, (we do content extraction first) we save quite a bit. Typically the content has much less markup than the template does.
We extract other things like the Opengraph image and social media links for author and store them independent of the content.
We extract other things like the Opengraph image and social media links for author and store them independent of the content.