Re:Memory efficiency of parallel IO operations in Python
by TickleSteve on Tuesday March 13, 2018 @08:39AM
(#16575614)
Small point, but stack-pointers are swapped... not the full stack.
The way you've worded it makes it seem like the whole stack is copied which of course is not the case.
[ Parent
| on HN ]
Re:Memory efficiency of parallel IO operations in Python
by jacobush on Tuesday March 13, 2018 @11:32AM
(#16576932)
True and in a machine with 0 cache, it's 100% true. Though, in real world with caches, it a little muddier. Yes, the stack pointer is switched. Depending on how much is referenced off that stack pointer, though, much of the stack may be in fact switched - in the cache.
Re:Memory efficiency of parallel IO operations in Python
by jstimpfle on Tuesday March 13, 2018 @09:45AM
(#16576007)
Right. And I think a typical stack size is ballpark 8MB, not 8KB, these days.
Re:Memory efficiency of parallel IO operations in Python
by scottlamb on Tuesday March 13, 2018 @01:12PM
(#16577700)
It varies. 8MiB is believable. That's 8MiB (plus a guard page) of address space. Typically it's divided into 4KiB ("small") pages which are only backed with real RAM once accessed. So maybe only 4KiB of real RAM, depending on what your program is doing...plus whatever bookkeeping the kernel needs, which I think can be as much as 128KiB on Linux, unfortunately. (At least something makes it use that much for me at work, although it might be related to something special I'm doing.)
It's also possible (but not common) to release accessed-but-no-longer-needed pages. Doing this when returning to a thread pool avoids situations where one occasional deep call chain causes all the threads in the pool to eventually become huge.
Re:Memory efficiency of parallel IO operations in Python
by Cyph0n on Tuesday March 13, 2018 @11:31AM
(#16576922)
I was working with pthreads recently and the default thread stack size is 8M.
Re:Memory efficiency of parallel IO operations in Python
by belovedeagle on Tuesday March 13, 2018 @11:29AM
(#16576904)
But it is: bringing the new stack into the cache, and eventually flushing the dirty one back to memory, is essentially the same as copying it.