Thanks jstoiko! You are right. Parallec is not specifically built to do crawling or parse website pages recursively (however you may build such crawler on top of it) We mostly use it to manage (HTTP/S) agents on every production machine in the cloud for software deployment, remediation, asset discovery etc. (Parallec like a kubenate master) to manage all the kubelet (agents)

Yes, we may or may not want to slow down. Sometimes if it is just regex or simple parsing we just put the parser inside of the worker. We can send the results out to Kafka etc so that some other process/machine can process them.

http://www.parallec.io/docs/submit-task/#apis-on-response-ha...