Python stands to lose its GIL, and gain a lot of speed

Re:Python stands to lose its GIL, and gain a lot of speed by lostdog on Sunday October 17, 2021 @01:54PM (#28898295)

Here's a use case: I was training a neutral net, and wanted to do some preprocessing (similar to image resizing, but without an existing C function). Inputs are batched, so the preprocessing is trivially parallelizable. I tried to multithread it in python, and got no speedup at all.

That was a really sad moment, and I've never felt good about python since.

[ Parent | on HN ]

Re:Python stands to lose its GIL, and gain a lot of speed by isoprophlex on Sunday October 17, 2021 @02:19PM (#28898499)

Yeah, I had something similar.

I wanted "as you wait for the GPU to churn through this batch, start reading the next batch from disk & preprocessing it on the CPU"

Getting this to work turned out so ass backwards it made me sad

Also I pity the fool who tries to connect a debugger to code using multiprocessing.Pool()...

[ Parent | on HN ]

Re:Python stands to lose its GIL, and gain a lot of speed by redeyed on Saturday October 23, 2021 @07:31AM (#28967451)

> Also I pity the fool who tries to connect a debugger to code using multiprocessing.Pool()

PyCharm works fine

[ Parent | on HN ]

Re:Python stands to lose its GIL, and gain a lot of speed by redeyed on Saturday October 23, 2021 @07:33AM (#28967461)

Yeah, had the exact problem.

I solved it by using python threads and C function (cv2.resize)

[ Parent | on HN ]