HPAT – A compiler-based framework for big data in Python

Re:HPAT – A compiler-based framework for big data in Python by The_Amp_Walrus on Friday October 13, 2017 @07:16PM (#15469823)

So this framework generates MPI compliant code, which is intended to be sent out to a bunch of super-computer nodes and executed.

What kind of existing infrastructure would you need to actually run this? I assume you would need to install agents that understand MPI on all your nodes and run some sort of centralised master agent to send out messages and supervise the whole thing. Is there a standard way to do this?

[ Parent | on HN ]

Re:HPAT – A compiler-based framework for big data in Python by jzwinck on Friday October 13, 2017 @08:29PM (#15470038)

Yes you need to install an MPI implementation such as OpenMPI or MPICH. You probably also want to install a scheduler such as TGSFKA Sun Grid Engine (https://en.m.wikipedia.org/wiki/Oracle_Grid_Engine), Torque, or Slurm. All of these contain a program called qsub which launches multiple copies of a job in parallel across machines. Perfect for MPI.

A vanilla compiler will suffice for MPI code, so you don't need anything special at build time.

[ Parent | on HN ]