Re:Linus Torvalds on Garbage Collection (2002)
by SpikeGronim on Friday April 22, 2011 @12:34PM
(#2474560)
The drawback with multiple processes is that they each have all the compiled bytecode on their heap. When I "import nltk" my heap goes from ~12 to ~36 MB. Add a few more dependencies and you end up wasting a non-trivial amount of RAM on python heaps.
[ Parent
| on HN ]
Re:Linus Torvalds on Garbage Collection (2002)
by lobster_johnson on Friday April 22, 2011 @12:56PM
(#2474628)
You could do the import and then fork the child processes. That should share all the memory used by the nitk module between all the children (for as long as the memory is not modified by anyone, which triggers a copy-on-write allocation of the affected pages).
most forks do copy-on-write and every access to a python object meddles with reference count, which is - you guessed it - a write.
Re:Linus Torvalds on Garbage Collection (2002)
by froydnj on Friday April 22, 2011 @01:31PM
(#2474747)
The context of the copy-on-write win was the bytecode for modules. I don't know that you'd have anybody meddling with the reference counts for that...but I haven't really looked.
Re:Linus Torvalds on Garbage Collection (2002)
by rayiner on Friday April 22, 2011 @10:52PM
(#2476128)
Bytecode is stored in function objects which IIRC are reference-counted.
Re:Linus Torvalds on Garbage Collection (2002)
by lobster_johnson on Saturday April 23, 2011 @04:03PM
(#2477458)
Aside from ad-hoc classes (defined within the scope of a function, say), does it ever make sense to garbage-collect a module or a class?
Say you do "import smtplib" in your main file. Now that module is imported -- forever. I don't know the internals of Python well enough, but I bet that the module reference has strong references to its contents, so that even if nobody is actually calling anything in smtplib, it will be there in case someone does. The same should be true about modules importing other modules; they stay visible at the scope-level, so they are permanently loaded.
So for those cases it would make sense to keep them separate from global garbage collection. I'm pretty sure that the method tables of all the classes in the system take up considerable space, probably in the order of megabytes for many apps.
Re:Linus Torvalds on Garbage Collection (2002)
by witten on Friday April 22, 2011 @12:56PM
(#2474629)
That's a valid point, but you can do things to mitigate this, like splitting up your program into processes that have different dependencies. For instance, if you have a UI process and a core logic process, you won't need to import your giant UI library twice.
Re:Linus Torvalds on Garbage Collection (2002)
by adambyrtek on Friday April 22, 2011 @05:38PM
(#2475511)
If you split the code horizontally instead of vertically you have to deal with IPC, which could be problematic (and slow). I would do that only if there was already a clearly defined interface between the modules, not just for the sake of concurrency. Moreover, taking into account growing number of cores in modern machines such strategy doesn't seem very future-proof.
Re:Linus Torvalds on Garbage Collection (2002)
by rayiner on Friday April 22, 2011 @10:49PM
(#2476126)
Or you could just use a language whose implementation doesn't blow.
Re:Linus Torvalds on Garbage Collection (2002)
by FooBarWidget on Saturday April 23, 2011 @04:02AM
(#2476472)
And suffer from 5x slower development time. Yeah your deadline now has to be extended from 2012 to 2015 but at least the language is FAST and is technically excellent, right?