Easy cluster parallelization with ZeroMQ

Re:Easy cluster parallelization with ZeroMQ by gh02t on Wednesday May 27, 2015 @03:52PM (#9613878)

Worth mentioning that the cluster tools in IPython implement basically the same system as he ends up with, plus lots of other functionality and it is using ZMQ as the backend as well. If you've got a small to medium sized task (I'd say less than 500 nodes is a good number) to run on your cluster ipcluster is pretty great. Even when I have a proper queuing system like SLURM or PBS in place, I still often use ipcluster on top.

http://ipython.org/ipython-doc/dev/parallel/

[ Parent | on HN ]

Re:Easy cluster parallelization with ZeroMQ by nickpsecurity on Wednesday May 27, 2015 @05:06PM (#9614346)

Thanks for the link. That might come in handy if I do any more clustering.

[ Parent | on HN ]

Re:Easy cluster parallelization with ZeroMQ by andreabedini on Thursday May 28, 2015 @12:05AM (#9616013)

I'm curious on how you use ipcluster on top of pbs

[ Parent | on HN ]

Re:Easy cluster parallelization with ZeroMQ by gh02t on Thursday May 28, 2015 @09:28AM (#9617957)

If you mean the practicalities of how to use it, they describe it a bit here http://ipython.org/ipython-doc/dev/parallel/parallel_process... .

If you mean how have I used it specifically, I've used it to do a variety of things. Ranging from recreating what was described in the original post to using it as an ad hoc MPI replacement. I find it's handy for managing jobs that have some coupling (and so aren't easily suited to running via the features of the batch system), but aren't tightly coupled enough to warrant the suffering of having to write MPI-based code.

[ Parent | on HN ]