News for hackers. Stuff that matters.
Re:How to write a crawler by drakaal on Sunday September 29, 2013 @08:35PM (#6467189)
For http://www.samuru.com we store our crawl in Google's DataStore. It gets big. By not storing all of the page, (we do content extraction first) we save quite a bit. Typically the content has much less markup than the template does.

We extract other things like the Opengraph image and social media links for author and store them independent of the content.

There are no comments on this comment yet.