There is a comment which was voted dead which actually answered the question. These services have their own crawlers. If you ever spot for example MajesticBot in your access logs you have found one of the biggest.