jeffpeiyt

The core of REST Commander is the actor based concurrency control described at http://www.ebaytechblog.com/2014/03/11/rest-commander-scalab... We recently open sourced the http://www.parallec.io/ to support more protocols and I suppose it is not hard to add new protocols if needed.

Thanks jstoiko! You are right. Parallec is not specifically built to do crawling or parse website pages recursively (however you may build such crawler on top of it) We mostly use it to manage (HTTP/S) agents on every production machine in the cloud for software deployment, remediation, asset discovery etc. (Parallec like a kubenate master) to manage all the kubelet (agents)

Yes, we may or may not want to slow down. Sometimes if it is just regex or simple parsing we just put the parser inside of the worker. We can send the results out to Kafka etc so that some other process/machine can process them.

http://www.parallec.io/docs/submit-task/#apis-on-response-ha...

Hi Uberneo, yes, we exactly use it for HTTP, but also others. here is a 20 lines example to extract from a lot of HTTPs to elastic search. https://github.com/eBay/parallec-samples/blob/master/sample-...

Parallec has a special super convenient response context let you pass in/out any object when handling the response. Now you can conduct scalable API calls, then pass aggregated data anywhere to elastic search, kafka, MongoDB, graphite, memcached, etc.

Python has global interpreter lock so if it is computational expensive, you have to use multi-process to use more than 1 core. Parallec can let the handler to run your onComplete() function either in worker before aggregation (parallel) or in manager after aggregation

Hi Michael, Thanks for the input! We will add evaluation on it. Starred it in github :-)

Thanks for your interests in REST Superman.

Is a great devOps tool to speak HTTP in scale.

Fast Parallel Async HTTP/REST/SOAP client as a service to monitor and manage 10,000 web servers. Sends requests to 1000 servers with response aggregation in 10 seconds. or 10,000 servers in 50 seconds.

Check details at http://www.restsuperman.com Fork it at https://github.com/eBay/restsuperman with zero installation! It is open source and totally free! Like me at facebook: https://www.facebook.com/getrestsuperman Follow me at twitter @restsuperman : https://twitter.com/restsuperman