Re:The GIL and its effects on Python multithreading
by bno1 on Wednesday September 29, 2021 @10:50AM
(#28694407)
In ML you often need to stream data from the training set because you can't hold it all in memory. Many times I noticed that my GPU would sit idle because python was busy loading more training examples. With proper multi threading you can have one or multiple threads loading data into a queue feeds the GPU thread. This is something extremely simple to do in any programming language that has doesn't have a GIL. And you can't just use a library for this, loading and preprocesding data is very specific to your application, and writing just this part in C completely defeats the purpose of choosing python in the first place.
[ Parent
| on HN ]
There are no comments on this comment yet.