Re:Solving multi-core Python
by semi-extrinsic on Friday July 10, 2015 @08:03AM
(#9863398)
> Issues generating random numbers - I ran 8M simulations on 8 cores, but because the random seed was forked, I actually only ran 1M simulations 8 times. (Easy enough to fix, but annoying.)
IIRC, even if you do your easy fix of giving the 8 threads different seeds, your results are not trustworthy since the parallell random number streams are likely to exhibit significant correlations. There's quite a bit of algorithm research behind quality PRNGs.
Collecting metrics require using an external collector process (statsd) rather than just an in-proc lib (e.g., codahale metrics on the JVM).
Issues generating random numbers - I ran 8M simulations on 8 cores, but because the random seed was forked, I actually only ran 1M simulations 8 times. (Easy enough to fix, but annoying.)
Deduplication is really handy in reducing memory usage - why would you want the same (immutable) object duplicated N times?
And of course various issues with resources - you need to be careful only to connect to the DB/etc after forking otherwise stuff gets weird. And of course, this means that instead of sharing a single connection pool, you've now got one DB connection per proc, which may or may not be a big waste of resources.