News for hackers. Stuff that matters.
Re:An introduction to parallel programming using Python's mu... by zobzu on Saturday July 12, 2014 @06:43PM (#8026007)
the author seems to be unaware of threading and event based models.

multiprocessing adds memory isolation through the CPU's protected memory.

Re:An introduction to parallel programming using Python's mu... by freyrs3 on Saturday July 12, 2014 @07:01PM (#8026045)
If you're working on CPU-bound tasks with NumPy/SciPy using threads then you have to think very hard to make sure most of the critical sections are hitting the NumPy calls in C which release the GIL. It's not a great reliable way to program. The way the author describes is basically the only pure-Python way of achieving parallelism for this kind of problem.
Re:An introduction to parallel programming using Python's mu... by zobzu on Saturday July 12, 2014 @07:10PM (#8026072)
it doesnt make it any less true. besides, that C python's GIL sucks doesnt mean threading or event models sucks.

In doubt, ask nginx how they feel about that. Then ask apache's mpn-prefork how that feels.

Re:An introduction to parallel programming using Python's mu... by freyrs3 on Saturday July 12, 2014 @07:14PM (#8026076)
If you're holding a global mutex every 100 instructions while context switching between CPU bound tasks, then yes, the GIL does suck. There are a class of IO-bound problems where threading/evented models in Python can be used effectively but that's not the class of problems the author is talking about here.
Re:An introduction to parallel programming using Python's mu... by nemothekid on Saturday July 12, 2014 @08:37PM (#8026220)
Considering the author's use case (mathematical modeling) and language (Python), threading and event-based models would have no real performance benefit.

Event based models only shine when you are doing IO-bound tasks. They won't help you when you are chewing CPU.

Threading models in Python aren't attractive because of the GIL. If you are doing a parallel matrix operation you can only ever use one CPU because of the GIL. Not attractive.

Re:An introduction to parallel programming using Python's mu... by chuckledog on Saturday July 12, 2014 @11:43PM (#8026576)
Any thoughts how software-transactional memory might apply for a use case like this?
Re:An introduction to parallel programming using Python's mu... by freyrs3 on Sunday July 13, 2014 @10:00AM (#8027343)
STM is not a great fit for this kind of problem, there's no need for all the transaction machinery if the problem is embarrassingly parallel. In an ideal world what you want is threads which just split the work sections like they do in C/C++/Haskell.
Re:An introduction to parallel programming using Python's mu... by rasbt on Saturday July 12, 2014 @06:52PM (#8026026)
Right. Yes, I am aware of the threading module. It depends on the type of application what to prefer: memory-bound vs CPU-bound tasks
Re:An introduction to parallel programming using Python's mu... by Igglyboo on Saturday July 12, 2014 @10:46PM (#8026472)
Just curious but could this be possible to get around by having multiple copies of the python interpreter installed on your sytstem? Maybe possibly changing some config strings so the GIL thinks it's something else.
Re:An introduction to parallel programming using Python's mu... by quadlock on Saturday July 12, 2014 @11:11PM (#8026510)
ProcessPoolExecuter from the futures package essentially does this, it runs multiple instances of the interpreter and then distributes instructions to them.