News for hackers. Stuff that matters.
Re:C++: A language for next generation web apps by jrockway on Wednesday January 27, 2010 @04:36AM (#1080205)
I don't get the constant complaints about the GIL. Letting your Python program run on 2 cores will make it 2x faster at best. Rewriting it in, say, Javascript or Lisp or Haskell or Java will make it run 2-50x faster on one core. After you get your 50x speedup, then you can worry about the 4x you'll get from buying 3 more processor cores.

(And oh yeah; it's only shared-memory concurrency that things like the GIL affects. If you have a job to do that wants to use 8 cores, split the job up into 8 parts and invoke your program 8 times. There's your 8x speedup.)

Re:C++: A language for next generation web apps by ramchip on Wednesday January 27, 2010 @04:53AM (#1080223)
> 2x faster at best

That's if you're CPU-bound. I don't use Python, but I made an image acquisition program in C++ which could be a relevant example. We wanted to save the images to disk in real-time (30-60FPS). Doing this in the acquisition loop would make the software unusable (the goal is video-rate confocal microscope imaging); it's far too long, and much of it is just due to disk writes being slow, not to the compression time. Using a thread pool was the solution, not because of an actual increase in speed, but because from the loop's POV the write went from blocking to non-blocking so the CPU stopped wasting time waiting for the disk.

We also wanted shared memory since there can be a lot of image data which is shared between the image compression & saving, display, and possibly statistics or filtering modules.

Re:C++: A language for next generation web apps by jrockway on Wednesday January 27, 2010 @12:29PM (#1080984)
Dunno about Python, but Perl has a library to do all disk writes in a separate (p)thread, so your main control thread never blocks on IO. (This is in addition to the usual event-loop tricks; I know Python can do nonblocking IO that way.)
Re:C++: A language for next generation web apps by fauigerzigerk on Wednesday January 27, 2010 @05:31AM (#1080246)
I can only tell you why _I_ am constantly complaining about the GIL. It's because I would like to use a Python/C combination for in-memory data analysis. C gives me the speed and memory efficiency and Python gives me the ease of use and the web stuff.

There is no 50x speedup to be had as it doesn't get any faster than C. The only significant speedup will come from parallelism. 8 cores this year, 16 next year and probably a 100 cores in a few years. Since I'm holding a lot of data in memory I can only run one process not many unless I implement each and every data structure on top of shared memory, which I'm not going to do because it's unproductive.

I cannot use Java or JavaScript or any language that doesn't have value types (i.e. structs and arrays of structs) with a well defined memory layout. I don't want to use Haskell because my problem doesn't lend itself to functional programming as it's inherently stateful. I feel I would have to fight the nature of Lisp to make it use as little memory as C. It makes no sense to use Lisp when I need to know how lists are laid out in memory.

The only realistic options right now are pure C, pure C++ or C#. Go does have all the right properties as well. It's very immature at this point though.

Re:C++: A language for next generation web apps by jbarham on Wednesday January 27, 2010 @10:11AM (#1080631)
Python C extension modules can release the GIL while they're running which allows for true concurrency. See this Google code search http://www.google.com/codesearch?q=python.org+lang%3Ac+Py_BE... for example usage.
Re:C++: A language for next generation web apps by fauigerzigerk on Thursday January 28, 2010 @04:09PM (#1085031)
I know this is possible. But I'm having trouble to imagine a web app design (or other network server) based on that idea. In order to use a lot of data in memory I would have to have a single Python process that gets called by nginx or apache. So that would be a bottleneck even before I get a chance to call my extension. And later there wouldn't be much code left that executes in the Python interpreter, which kind of defeats the purpose.

But I have to admit that I haven't fully thought this possibility through. Maybe you're right that it can be made to work.