News for hackers. Stuff that matters.

A comment on Ask HN: Has anyone ever crawled over a billion pages? How much did it cost?. Parent comment

Re:Ask HN: Has anyone ever crawled over a billion pages? How... by AznHisoka on Wednesday September 14, 2016 @11:12AM (#12497731)
Ruby and Sidekiq as the messaging queue

Postgres to store the data

Elasticsearch as a search index.

My ES cluster has around 10 nodes, 64 GB RAM, quad-core.

Postgres database cluster is 4 nodes, 1 TB, 64 GB RAM, quad-core.

800 crawler threads distributed across 10 dedicated servers.

[ Parent | on HN ]
Re:Ask HN: Has anyone ever crawled over a billion pages? How... by outpan on Wednesday September 14, 2016 @01:26PM (#12499243)
Thanks a lot! This sounds reasonable. Did you guys look into professional services for this?
[ Parent | on HN ]
Re:Ask HN: Has anyone ever crawled over a billion pages? How... by AznHisoka on Wednesday September 14, 2016 @04:46PM (#12501063)
Nope. We have lots of custom needs.
[ Parent | on HN ]