Parallel Python

Re:Parallel Python by rev_d on Tuesday March 14, 2017 @09:54AM (#13867458)

Wouldn't it make a lot of sense to just use Pyspark with RDDs? Latency would be relatively high, but it'd also bypass the GIL while also being more modern.

[ Parent | on HN ]

Re:Parallel Python by mangecoeur on Tuesday March 14, 2017 @10:10AM (#13867597)

In my experience pyspark is much more flaky and annoying that doing parallel computing with more 'python native' tools. It only really makes sense when you outgrown small clusters and really need huge infrastructure.

[ Parent | on HN ]

Re:Parallel Python by splike on Tuesday March 14, 2017 @10:15AM (#13867626)

What python tools do you use for small clusters?

[ Parent | on HN ]

Re:Parallel Python by elyase on Tuesday March 14, 2017 @01:49PM (#13869756)

Dask would be an option.

[ Parent | on HN ]

Re:Parallel Python by mangecoeur on Wednesday March 15, 2017 @04:08AM (#13874560)

Was going to say that. Or ipython parallel if you want to go lower level

[ Parent | on HN ]