I did a personnal webcrawler using PHP, Redis, Gearman on a single (personnal) computer with many VMs to emulate AWS instances and it works great ! You can surely improve this by using other technologies than PHP (python, C, nodejs) and Gearman (Kafka, rabbitmq).
You can start reading this article about the BFS algorithm : https://fr.khanacademy.org/computing/computer-science/algori...
I did a personnal webcrawler using PHP, Redis, Gearman on a single (personnal) computer with many VMs to emulate AWS instances and it works great ! You can surely improve this by using other technologies than PHP (python, C, nodejs) and Gearman (Kafka, rabbitmq).
Hope this helps