News for hackers. Stuff that matters.

A comment on Ask HN: Where do backlink checking services get their data from?.

Re:Ask HN: Where do backlink checking services get their dat... by tconaugh on Tuesday August 22, 2017 @04:47PM (#15076234)
They use distributed web crawlers to crawl 100s of billions of web pages. Probably one of the following options:

1) Built their own crawlers.

2) Using an Apache Nutch/Heritrix cluster in a colo facility.

3) Use 3rd party services like mixnode.

[ Parent | on HN ]

There are no comments on this comment yet.