News for hackers. Stuff that matters.
Re:PyParallel: How we removed the GIL and exploited all cores by trentnelson on Saturday June 07, 2014 @08:48AM (#7861967)
If you want to see me try get through 153 slides in 45 minutes (and fail), this talk was recorded here: http://vimeo.com/79539317.

There's a more recent but slightly shorter deck that focuses more on the general concurrency/parallelism problem here: https://speakerdeck.com/trent/parallelism-and-concurrency-wi...

Re:PyParallel: How we removed the GIL and exploited all cores by NelsonMinar on Saturday June 07, 2014 @11:37AM (#7862311)
What would be terrific is a 400 word article summarizing what you did.
Re:PyParallel: How we removed the GIL and exploited all cores by darkseas on Sunday June 08, 2014 @02:12AM (#7864104)
There is a fair summary written by Jonathon Allen on InfoQ [0]

[0] http://www.infoq.com/articles/PyParallel

Re:PyParallel: How we removed the GIL and exploited all cores by trentnelson on Sunday June 08, 2014 @01:21PM (#7865230)
I inadvertently ran into this article a few weeks ago and was pleasantly surprised... it's not a bad summary at all (especially considering I had no contact with Jonathan).
Re:PyParallel: How we removed the GIL and exploited all cores by jnazario on Saturday June 07, 2014 @10:34AM (#7862175)
> 153 slides in 45 minutes (and fail)

less is more :) one one the best and most memorable talks i went to was 4 slides long.

that said i've been thinking a lot the past week or so about a machine i can dedicate to scientific computing, preferably using Sage noteboks or something like that. i have some long running evaluations (graphs) that i would rather not crush my laptop for. and so i get to thinking about the kind of machine i want to do this on, something parallel is tempting because of the higher speed gains. that said, i'm wondering if that would help Python (and hence Sage) or not. so now i'm looking at something like iJulia (Julia in an IPython notebook) and cobbling together something like this.

than i see your parallel python mods and wonder if i should try this. should i? will it help me in my use case (speeding up a Sage server)?

Re:PyParallel: How we removed the GIL and exploited all cores by turnersr on Saturday June 07, 2014 @11:11AM (#7862260)
My guess is that you can't drop in PyParallel without any work of porting Sage because it's not clear how you would pass objects around like those used for GAP. Sage also would have to be rewritten to use this new API of implementing protocol-based classes.

You probably don't want to use all of Sage for that anyway because you are most likely not using GAP, for example. Instead gut out parts of Sage and look at IPython's interface to Spark http://nbviewer.ipython.org/gist/JoshRosen/6856670 or implement a protocol for your specific computation and use PyParallel.

I highly doubt you just point to new a Python implementation and tell Sage to just roll with it and expect things to be automagically faster.

Also, if you have a long running job, then why would you do it in a notebook that requires you to keep your browser open?

Re:PyParallel: How we removed the GIL and exploited all cores by trentnelson on Saturday June 07, 2014 @07:10PM (#7863402)
^^^ what turnersr said.

(Spot on, by the way.)

Re:PyParallel: How we removed the GIL and exploited all cores by jnazario on Saturday June 07, 2014 @12:38PM (#7862426)
thanks, turnersr, all good points.
Re:PyParallel: How we removed the GIL and exploited all cores by vit_tucek on Saturday June 07, 2014 @02:29PM (#7862675)
I use cloud.sagemath.com for offloading heavy computational work.
Re:PyParallel: How we removed the GIL and exploited all cores by GFK_of_xmaspast on Saturday June 07, 2014 @02:18PM (#7862646)
Depending on what kind of graph problem you might consider graphlab or one of the other specialty graph processing systems out there.
Re:PyParallel: How we removed the GIL and exploited all cores by ska on Saturday June 07, 2014 @10:50AM (#7862212)
Fwiw, 2min/slide (averaged) isn't a bad heuristic to start with.
Re:PyParallel: How we removed the GIL and exploited all cores by bayesianhorse on Saturday June 07, 2014 @10:00AM (#7862110)
If you don't have time for 153 slides, maybe you shouldn't have 153 slides in the deck ;-)

"Completeness" isn't really a good argument against shortening the talk ruthlessly. I watched the video and at this speed I hardly think I remember more than maybe 10 slides' worth anyway.

Re:PyParallel: How we removed the GIL and exploited all cores by sp332 on Saturday June 07, 2014 @11:50AM (#7862330)
But we wouldn't know that if he hadn't tried :)