Ask HN: Where do backlink checking services get their data from?

Posted by maurtinshkreli on Tuesday August 22, 2017 @04:40PM
from the ask-hn dept.

I'm talking about Moz, ahrefs, megaindex and the like...

( 2 comments | 8 points | on HN )

Re:Ask HN: Where do backlink checking services get their dat... by sebst on Tuesday August 22, 2017 @05:06PM (#15076393)

There is a comment which was voted dead which actually answered the question. These services have their own crawlers. If you ever spot for example MajesticBot in your access logs you have found one of the biggest.

[ Parent | on HN ]

Re:Ask HN: Where do backlink checking services get their dat... by tconaugh on Tuesday August 22, 2017 @04:47PM (#15076234)

They use distributed web crawlers to crawl 100s of billions of web pages. Probably one of the following options:

1) Built their own crawlers.

2) Using an Apache Nutch/Heritrix cluster in a colo facility.

3) Use 3rd party services like mixnode.

[ Parent | on HN ]