News for hackers. Stuff that matters.
Re:Let's Remove the Global Interpreter Lock by detroitcoder on Wednesday August 16, 2017 @12:00PM (#15028534)
DAMN. I just did a basic test and it kinnda just worked?!? I created a test dataframe of 100M rows X 10 cols which took up ~2.3G and then used joblib.dump within the on_starting hook which is run when the gunicorn master starts up. Then loaded that df in with joblib.load within the worker and the total memory consumption was practically flat. Then I bumped up the number of workers to 20 and still flat. That is actually amazing. Coolest thing I have seen in months for how easy it is. Now I have to test out if the analytics actually work and a deep dive into the mechanics of mem-mapping.
Re:Let's Remove the Global Interpreter Lock by ogrisel on Thursday August 24, 2017 @08:31AM (#15089518)
Thanks for your feedback. I am glad I could help you.