How to write a crawler

Re:How to write a crawler by gondo on Sunday September 29, 2013 @11:59PM (#6467700)

why do you need Cassandra? isn't it possible to use different storage engines for ES? (disk + memory)

[ Parent | on HN ]

Re:How to write a crawler by cmiles74 on Monday September 30, 2013 @08:18AM (#6469110)

I was storing a lot of raw data (the content of the page crawled, PDF files) and there's no real gain to keeping it inside of the index.

[ Parent | on HN ]

Re:How to write a crawler by gondo on Monday September 30, 2013 @07:43PM (#6473069)

hm im doing similar thing but im storing raw data in S3 and glacier. no need for nosql for this. or am i missing something?

[ Parent | on HN ]