News for hackers. Stuff that matters.
Re:BrainFuck inspired scheduler successfully replaced the Py... by janzer on Friday March 26, 2010 @12:58AM (#1220296)
To clarify a little, this patch does not eliminate the GIL it just schedules the next thread to acquire the GIL using the BFS scheduling algorithm.

Also and perhaps more importantly this has not been incorporated into any version of python. It is just a patch on the bug tracker and realistically I doubt it has much chance of being accepted.

Re:BrainFuck inspired scheduler successfully replaced the Py... by kinghajj on Friday March 26, 2010 @01:09AM (#1220316)
If he's fixed the bugs and gotten it to be portable (at least to POSIX), then why shouldn't it get adopted? Just look at those benchmark results! 259ms per loop instead of the next best of 1.25s. If a real app gets improvements from this, it should be a no-brainer.
Re:BrainFuck inspired scheduler successfully replaced the Py... by mbreese on Friday March 26, 2010 @01:20AM (#1220326)
I'd wait to see how this works with a real workload as opposed to his contrived example. It looks like it might be good, but it definitely needs to be better tested.
Re:BrainFuck inspired scheduler successfully replaced the Py... by sid0 on Friday March 26, 2010 @03:39AM (#1220499)
Glancing at the patch, it looks like it should work on both POSIX and Win32.
Re:BrainFuck inspired scheduler successfully replaced the Py... by JoachimSchipper on Friday March 26, 2010 @05:44AM (#1220627)
At least the patch as first proposed suffered from all-the-world-is-Linux (see the discussion of CLOCK_THREAD_CPUTIME_ID)...
Re:BrainFuck inspired scheduler successfully replaced the Py... by rbanffy on Friday March 26, 2010 @06:19AM (#1220669)
You have to start with one platform. You then make it work on the others.

If I write something that initially only runs on IRIX and then make it work on other OSs, it would be unfair to attribute it an all-the-world-is-IRIX. It just happens the guy had a Linux box to refine his idea to the point of working, unfortunately using stuff other OSs don't offer in the same way.

Re:BrainFuck inspired scheduler successfully replaced the Py... by prewett on Friday March 26, 2010 @11:12AM (#1221186)
It was based off a piece of the Linux kernel... I'd say it isn't so much an all-the-world-is-Linux so much as Linux is what is available.
Re:BrainFuck inspired scheduler successfully replaced the Py... by jrockway on Friday March 26, 2010 @04:51AM (#1220573)
Assuming it fixes your app, instead of using "python" to run it, you just start using his version. The joys of open source.
Re:BrainFuck inspired scheduler successfully replaced the Py... by rbanffy on Friday March 26, 2010 @06:14AM (#1220664)
The problem then is that you now have to maintain your own fork of Python.

Again, the joys of open-source ;-)

Re:BrainFuck inspired scheduler successfully replaced the Py... by jrockway on Friday March 26, 2010 @09:00AM (#1220864)
This is easier than it sounds.
Re:BrainFuck inspired scheduler successfully replaced the Py... by jerf on Friday March 26, 2010 @10:17AM (#1221052)
Well, it would be easier than it sounds but a bit of Googling reveals that while Python plans on moving to Mecurial, it's still in SVN.

Still, git-svn can solve that problem pretty well.

Re:BrainFuck inspired scheduler successfully replaced the Py... by rbanffy on Friday March 26, 2010 @10:51AM (#1221138)
If it's easy, chances are many people will participate and the fork will take on a life of its own. If not, you will have to merge back the patch from time to time.

It's not a terrible thing, if the parts the patch changes don't get changed often. If they do, it could quickly turn into a nightmare

Re:BrainFuck inspired scheduler successfully replaced the Py... by alnayyir on Friday March 26, 2010 @09:13AM (#1220888)
I program in Python (Django) full-time. It's my job. I'm the only developer at my company who knows C.

I'm sorry but mere mortals who just want to get stuff done are not going to maintain their own fork of an entire programming language. I don't know what alternative reality you've been spending your time in (academia? FSF employee?) but it's simply not going to occur unless someone else takes up the burden and makes regular public releases.

Notice that usage of CK patches by Linux users evaporated as soon as he went on software sabbatical.

Open source is not driven by crowd sourcing or collective effort, it's driven by a small number of heroes willing to devote time in order to benefit everyone else.

Re:BrainFuck inspired scheduler successfully replaced the Py... by timwiseman on Friday March 26, 2010 @01:05PM (#1221483)
Open source is not driven by crowd sourcing or collective effort, it's driven by a small number of heroes willing to devote time in order to benefit everyone else.

I agree completely.

It is worth noting that this is true of mast crowd sourced projects though. A small cadre of devotees put in heroic effort and then a larger group contributes now and then, and everyone else either contributes nothing or some money.

Re:BrainFuck inspired scheduler successfully replaced the Py... by hartror on Friday March 26, 2010 @04:55AM (#1220576)
Wow I was excited and confused for a second, I thought the impossible had been done! That's fine that it hasn't I am happy with Twisted Spread for multi-process RPC needs.
Re:BrainFuck inspired scheduler successfully replaced the Py... by ash on Friday March 26, 2010 @03:23AM (#1220474)
The title is wrong. Brain Fuck scheduler is not related to brainfuck language.
Re:BrainFuck inspired scheduler successfully replaced the Py... by mahmud on Friday March 26, 2010 @01:05AM (#1220310)
The scheduler is a simplified implementation of the recent kernel Brain Fk Scheduler by the Linux hacker Con Kolivas

Not as fun now is it? Kolivas is a leader on scheduling, he can attribute his hacks to whatever joke language out there and that wont make them any less stellar.

Re:BrainFuck inspired scheduler successfully replaced the Py... by dschobel on Friday March 26, 2010 @02:29AM (#1220400)
I had totally missed Con's return from self-imposed exile. Very exciting. I loved his previous scheduler work. Glad to see him back at it.
Re:BrainFuck inspired scheduler successfully replaced the Py... by viraptor on Friday March 26, 2010 @12:59AM (#1220297)
As far as I understand, it just changes the way thread scheduling works, but doesn't make Python "properly multithreaded". That means it's still only one active non-native-extension thread running at any time. Could someone confirm it?

Edit: I guess janzer confirmed this at posting at the same time.

Re:BrainFuck inspired scheduler successfully replaced the Py... by jey on Friday March 26, 2010 @01:01AM (#1220301)
No, Python doesn't work like that even with the traditional GIL. The problem is that even when you have multiple OS threads, they all end up competing for the same lock which kills throughput (but not as badly as the fully-serialized scenario you described). By using a scheduler the locking order can at least be controlled a bit more to improve throughput. [As far as I know; been a few years since I dug through CPython.]
Re:BrainFuck inspired scheduler successfully replaced the Py... by viraptor on Friday March 26, 2010 @01:25AM (#1220332)
What do you mean by "not as badly as the fully-serialized scenario"? I thought that Python threads are fully serialized, apart from extensions code which can spawn their own threads and release the GIL during operations that don't affect the python memory (IO mainly). Interpreter still switches Python threads using GIL, but the Python code itself never runs in parallel.

Are we talking about the same thing, or is there some other non-serialized scenario?

Re:BrainFuck inspired scheduler successfully replaced the Py... by jey on Friday March 26, 2010 @01:45AM (#1220353)
I'm pretty sure that there's a good chunk of stuff you can do in Python without acquiring the GIL -- the problem is that in practice you end up doing a lot of I/O and stuff that requires at least momentarily acquiring the GIL, leading to contention. So if you stuck to the operations that didn't require locking any GIL-protected data, you could run at full throughput. It's at least not the case that the GIL is held all the time while running a Python thread -- the problem is instead that your threads end up having to acquire it often.
Re:BrainFuck inspired scheduler successfully replaced the Py... by JoachimSchipper on Friday March 26, 2010 @05:47AM (#1220628)
Actually, the GIL is needed to execute Python code (well, access Python objects). It is released by I/O- or computation-heavy C code, so e.g. SciPy or reading files allows some level of parallelism, but pure-Python code will be serial.
Re:BrainFuck inspired scheduler successfully replaced the Py... by devinj on Friday March 26, 2010 @12:35PM (#1221407)
Pure Python code can do a lot. Large parts of the stdlib are extension modules, and can release the GIL (e.g. one of the tests on the bug tracker used time.sleep()). In practice, if your code is IO-bound, it's doing work in an extension module, and if it's CPU-bound, it should be doing its work in an extension module. So there's actually not a huge problem.
Re:BrainFuck inspired scheduler successfully replaced the Py... by jey on Friday March 26, 2010 @02:16PM (#1221661)
I stand corrected. And frightened. fork(), here I come!
Re:BrainFuck inspired scheduler successfully replaced the Py... by cma on Friday March 26, 2010 @08:18PM (#1222339)
fork() isn't that great for a lot of situations. If you are thinking of taking advantage of your operating system's copy-on-write paging by loading a large chunk of data to be used read-only, forking a bunch of processes, processing the data each processes, and finally, 'reducing' the results of all of the forks into some sort of output, don't bother.

What happens is when you read an object in one process, python increments the reference count, thus touching the memory page, thus copying it, thus screwing you.

(however, compacting garbage collection turns out to have more or less the same problem)

Re:BrainFuck inspired scheduler successfully replaced the Py... by Snark7 on Friday March 26, 2010 @12:42AM (#1220284)
This is related to Python 3.2 only. In other words, this is not noteworthy.
Re:BrainFuck inspired scheduler successfully replaced the Py... by wisty on Friday March 26, 2010 @01:02AM (#1220303)
Python is python. I wouldn't be surprised to see it ported to python 2.7 if it does work. At the moment I doubt it's production ready - lots of testing and validation before it goes live. I thing Guido explicitly said that removing the GIL is the sort of thing he would like to see in Python 2.X.
Re:BrainFuck inspired scheduler successfully replaced the Py... by apgwoz on Friday March 26, 2010 @05:55AM (#1220638)
Actually, though I'm using 2.X in everything I'm doing, I'd rather see it only appear in 3.X. There has to be something that drives people to port stuff to 3.X or it's not going to happen. Dramatic speed improvements such as what this potentially provides would be extremely helpful in that regard. The other hope right now, is of course unladen swallow, which hasn't proved to be very significant yet, as far as I'm concerned.
Re:BrainFuck inspired scheduler successfully replaced the Py... by sapphirecat on Friday March 26, 2010 @08:51AM (#1220843)
3.X can be pretty awesome, but as long as projects want to maintain compatibility with 2.5 or earlier, it's going to be difficult to get some serious porting momentum going. Once 2.6+ becomes a practical development target, 3.x will be a much easier sell.

At least, that's my perspective after watching PHP 5 slowly catch on amongst PHPers, even though it had many more improvements (e.g. objects are no longer value types), and far fewer compatibility breaks.

Re:BrainFuck inspired scheduler successfully replaced the Py... by jrockway on Friday March 26, 2010 @09:03AM (#1220869)
People are lazy. I still hear people wanting Perl 5.8 compatibility for my modules, even though 5.10 is 2 years old and has 100% backwards and forwards compatibility with Perl 5.8. In other words, all your existing code will run unmodified, and any 5.10-specific features you use will cause 5.8 to die at compile time.

People confuse me.

Re:BrainFuck inspired scheduler successfully replaced the Py... by astine on Friday March 26, 2010 @09:59AM (#1220997)
I use a lot of Perl 5.8 at the LoC because it's the only dynamic language that comes installed by default on Solaris 10. That, and because it is the primary language of a proprietary product that we have to use. I'd love to use Perl 5.10, but then I'd have to install it on all of the machines on which my code is expected to run. If I had that kind of control, I'd skip Perl and go straight to Python or Ruby. (Actually, that was a lie, I'd use Common Lisp if I could.) As it is, I've standardized on Perl 5.8.
Re:BrainFuck inspired scheduler successfully replaced the Py... by jrockway on Friday March 26, 2010 @04:50PM (#1222042)
They can install your product, but not if you bundle Perl/Python/Ruby in that directory?
Re:BrainFuck inspired scheduler successfully replaced the Py... by astine on Friday March 26, 2010 @07:26PM (#1222280)
The product I am talking about is Signiant:http://www.signiant.com/. It's a file based workflow application that basically that is written in Perl in the same sense that emacs is written in emacs lisp. The idea is for people to write workflows in the embedded Perl environment which is the same across of the machines on which Signiant is installed. I could use another interpretor, but that would require extra work and I wouldn't be able to use a lot of the Signiant specific code.