Ask HN: Has anyone ever crawled over a billion pages? How much did it cost?

Re:Ask HN: Has anyone ever crawled over a billion pages? How... by AznHisoka on Tuesday September 13, 2016 @08:28PM (#12493394)

I've crawled over a billion pages over a stretch of 3 years or so. Crawling is the easy task and just crawling a billion pages wouldn't cost more than a few thousand a month. Add a couple more thousand for storing these pages in a search index and database.

[ Parent | on HN ]

Re:Ask HN: Has anyone ever crawled over a billion pages? How... by theviajerock on Wednesday September 14, 2016 @07:49PM (#12502243)

Do you have a company that does this ? Can you advice me about this? I like the crawling thing, I would like to know how to monetize this.

Thanks!

[ Parent | on HN ]

Re:Ask HN: Has anyone ever crawled over a billion pages? How... by outpan on Tuesday September 13, 2016 @08:37PM (#12493434)

Would you be able to share what your stack was? and the resources it took? Thanks a lot.

[ Parent | on HN ]

Re:Ask HN: Has anyone ever crawled over a billion pages? How... by AznHisoka on Wednesday September 14, 2016 @11:12AM (#12497731)

Ruby and Sidekiq as the messaging queue

Postgres to store the data

Elasticsearch as a search index.

My ES cluster has around 10 nodes, 64 GB RAM, quad-core.

Postgres database cluster is 4 nodes, 1 TB, 64 GB RAM, quad-core.

800 crawler threads distributed across 10 dedicated servers.

[ Parent | on HN ]

Re:Ask HN: Has anyone ever crawled over a billion pages? How... by outpan on Wednesday September 14, 2016 @01:26PM (#12499243)

Thanks a lot! This sounds reasonable. Did you guys look into professional services for this?

[ Parent | on HN ]

Re:Ask HN: Has anyone ever crawled over a billion pages? How... by AznHisoka on Wednesday September 14, 2016 @04:46PM (#12501063)

Nope. We have lots of custom needs.

[ Parent | on HN ]

Re:Ask HN: Has anyone ever crawled over a billion pages? How... by usgroup on Wednesday September 14, 2016 @06:18AM (#12495460)

Just in case you don't know common-crawl makes available a huge crawl dataset

[ Parent | on HN ]

Re:Ask HN: Has anyone ever crawled over a billion pages? How... by outpan on Wednesday September 14, 2016 @01:27PM (#12499252)

Common Crawl is great! however, some use cases require larger crawls with a higher frequency.

[ Parent | on HN ]