Re:The multiprocessing module in Python
by Symmetry on Friday February 03, 2017 @03:45PM
(#13563148)
A memory consumption thing that you have to be careful of when doing that that we ran into recently. When you call fork Linux doesn't actually have to copy any of the existing memory until it's changed thanks to the magic of copy-on-write. But often Python will quickly increment or decrement the reference counters on all those bits of memory forcing copies; increasing memory usage and slowing things down. So either being careful with that or turning off garbage collection for the process (if it's short lived) are things to consider there.
[ Parent
| on HN ]
Re:The multiprocessing module in Python
by jsmeaton on Friday February 03, 2017 @04:36PM
(#13563542)
This was discussed recently in an instagram blog post. Did your team also independently reach these conclusions?
Re:The multiprocessing module in Python
by Symmetry on Friday February 03, 2017 @06:46PM
(#13564420)
I'm not sure what the order of events was but I should certainly link that blog post.
https://engineering.instagram.com/dismissing-python-garbage-...