tconaugh
Re:Ask HN: Where do backlink checking services get their data from?
on Tuesday August 22, 2017 @04:47PM (#15076234)
They use distributed web crawlers to crawl 100s of billions of web pages. Probably one of the following options:
1) Built their own crawlers.
2) Using an Apache Nutch/Heritrix cluster in a colo facility.
3) Use 3rd party services like mixnode.