News for hackers. Stuff that matters.
Re:The Python GIL Visualized by thorax on Tuesday January 05, 2010 @12:44PM (#1032794)
Sure, except when that doesn't make any sense.

I embed Python in my application. My highly graphical application requires XY,000 KB of memory. It's probably not wise to re-invoke my entire app Z times simply so that Python can properly take advantage of the user's multi-core OS while scripting inside my application.

Any time you embed Python (e.g. a game, a browser, etc), the multiprocessing support isn't going to work very well as it is today. Ideally we get to the point where threading works great in the core interpreter.

I asked Guido about this in front of his keynote audience a few Pycons ago, and his answer was a sad shrug and saying to use Jython or IronPython instead.

I'd love to see the GIL improved/removed because CPython-embedding apps don't have the same options as everyday pure scripting.

Re:The Python GIL Visualized by scott_s on Tuesday January 05, 2010 @02:01PM (#1032962)
I agree with you, but in practice, there won't be as much memory replication as you assume. Linux (and I assume most modern OSes) implement copy-on-write: no data is copied until a write occurs. So as long as the separate process only read that large data set, that large data set should exist in memory only once.
Re:The Python GIL Visualized by FooBarWidget on Tuesday January 05, 2010 @04:35PM (#1033443)
Exception multiprocessing does not use fork() (without exec()) because it has to be compatible with Windows.
Re:The Python GIL Visualized by rbanffy on Tuesday January 05, 2010 @06:39PM (#1033708)
Like I use to say: that's Microsoft holding everybody back. ;-)

Now, seriously, can we implement a better multiprocessing module for those with modern OSs and leave a less functional one (one that will raise a LesserOSError when certain functionality is invoked)?

Re:The Python GIL Visualized by jnoller on Tuesday January 05, 2010 @10:06PM (#1034047)
Incorrect; On linux and OS/X it uses fork. On windows it uses a best-effort clone.
Re:The Python GIL Visualized by malkia on Tuesday January 05, 2010 @02:16PM (#1033014)
In case with Lua - I can simply create new context, and voila a whole new lua vm is there. Not sure whether that could be done with embeddable python.
Re:The Python GIL Visualized by est on Tuesday January 05, 2010 @03:06PM (#1033173)
I guess because python vm is 2MB while Lua is just hundereds of kilobytes
Re:The Python GIL Visualized by codexon on Tuesday January 05, 2010 @04:24PM (#1033411)
Does Lua also have a GIL?
Re:The Python GIL Visualized by silentbicycle on Tuesday January 05, 2010 @04:36PM (#1033448)
No, it doesn't. The whole interpreter was designed from the beginning to be embedded in another (typically C or C++) project, and they didn't want to have any concurrency infrastructure in the core language that would clash with the surrounding project's design. (It has excellent support for coroutines, though.)

Rather than having static variables and a GIL, the interpreter keeps all its state behind an opaque pointer to a VM context. The VM is quite lightweight (a few hundred K), and running in each thread or process is not a big deal. With forking, you even share the standard libraries. In practice, this just means that every call to the Lua C API starts with ("lua_State *L"). Not a big deal.

Lua also has real tail-calls, closures, and lambdas. It's a great language. Rather like a smaller, cleaner, less opinionated Python.

Re:The Python GIL Visualized by malkia on Tuesday January 05, 2010 @06:12PM (#1033662)
Also lua right now has the one of the best JIT's out there (luajit). The guy behind the project is simply genius. Really good stuff. And also kept small. It's x2-x3 faster than V8 on specific benchmarks that I've did (mainly dealing with 3d model meshes and modifying them).

It's only for x86 right now, but he's working on x64 port.

Re:The Python GIL Visualized by silentbicycle on Tuesday January 05, 2010 @07:09PM (#1033762)
LuaJIT is really impressive, but honestly, the stock interpreter is fast enough for me in general, and I'm fine with just moving hotspots to C as needed.

I'm more impressed with how concise Lua is. I've got the Lua 5.1 Reference Manual sitting in front of me. It's 103 pages for the whole language (roughly half of which is the C API), the entire syntax fits on page 95, etc. I can keep it all in my mental L1 cache, so to speak. Yet, it's expressive enough to have displaced Python as my go-to language. It's not perfect, but I'm very happy with the trade-offs it makes.

I'll be psyched once it's ported to a couple more hardware platforms, though.

Re:The Python GIL Visualized by codexon on Tuesday January 05, 2010 @10:30PM (#1034083)
I just read the Lua manually, and it appears as though Lua has a GIL.

http://lua-users.org/wiki/ThreadsTutorial

ANSI C, the standard to which Lua complies, has no mechanism for managing multiple threads of execution.

The method above for implementing threading in Lua using a global mutex is inefficient on multiprocessor systems. As one thread holds the global mutex, other threads are waiting for it. Thus only one Lua thread may be running at a time, regardless of the number of processors in the system.

Am I misinterpreting what's being written here?

Re:The Python GIL Visualized by silentbicycle on Tuesday January 05, 2010 @10:54PM (#1034125)
It's a bit confusing. Essentially, "the method above" (using multiple OS threads running on the same Lua VM) extends Lua, adding a GIL along the way.

Lua itself does not use pre-emptive threads (the kind where you need to worry about locks, race conditions, "thread safe" libraries, etc.). It has co-routines, co-operative threads, in which independent light processes run and yield to each other. Only one is running in the interpreter at a time, so locking and race conditions are much less an issue. If something needs to happen atomically, you just don't yield and resume in the middle of it. (Coroutines are as versatile as one-shot continuations, incidentally - each can be implemented in terms of the other.)

BUT, since Lua is designed for use in other projects, it may be running as a scripting engine inside of (say) a multi-threaded C++ game. In this case, Lua can have multiple threads running in the VM at once. There are no-op functions (lua_lock and lua_unlock) defined in the code around operations which need to be atomic. Such a project could recompile Lua with them calling appropriate locking operations. You can run Lua multithreaded, if you want, but then you have to worry about thread safety. (I have to admit some ignorance here, though - I think running multiple independent VMs is much simpler than having to deal with debugging multi-threaded code.)

FWIW, the source for Lua 5.1.4 is ~626 KB. Adding a customized version of it to a project's source tree is nowhere near a big deal as including, say, all of Python (which is, what, 40+ MB?). In the Lua source, luaconf.h collects some items that are likely to be tweaked in project-specific ways.

Anyway, Lua does not have any threading implementation of its own because it would make it far more difficult to run inside of other projects. The Lua VM itself is pretty light (a few hundred KB), and since all VM state is specific to each instance, you can run several without running into the sort of problems that people have with Python's GIL.

There are several extensions for multitasking in Lua, beyond using coroutines. Here is a listing on the wiki (http://lua-users.org/wiki/MultiTasking) and a comparison by the author of Lanes (http://kotisivu.dnainternet.net/askok/bin/lanes/comparison.h...).

This is one of the cases where keeping Lua small, portable, and easy to embed means that functionality people expect ends up in an extension rather than as part of the "official" standard library. I'm happy with it as a trade-off - it makes Lua as expressive as Python but keeps the core language as small and clean as SQLite.

Re:The Python GIL Visualized by kilowatt on Tuesday January 05, 2010 @01:29PM (#1032887)
Just because your frontend uses XY,000 KB of memory doesn't mean that worker processes have to too. The working set size for a fresh Python 2.6 run on my Windows 7 box is 6,620KB. See Chrome for a good example of this approach :)
Re:The Python GIL Visualized by tentonova2 on Tuesday January 05, 2010 @01:34PM (#1032894)
Chrome's approach was not chosen because of its general purpose suitability as a multi-core processing model, but because of the security advantages of relying on OS process separation in a web browser.

Chrome is a very specific use case with unusual requirements. If you take anything away from the Chrome example, it would be that securely replicating the functionality easily available from shared-memory threads in a multi-processing model is both very complicated and not very portable.

Re:The Python GIL Visualized by thorax on Tuesday January 05, 2010 @01:52PM (#1032935)
We're talking about the multiprocessing package/modules in Python:

http://docs.python.org/library/multiprocessing.html

It uses fork under the covers, so small worker processes aren't an option for that module (unless we forego interpreter embedding entirely which isn't the point).

There are other techniques for spawning processes to do some external process work, but those, of course, need entirely separate plumbing.