Re:How to choose the right Python concurrency API
by DrScientist on Wednesday August 10, 2022 @05:09AM
(#32409444)
Queues.
If you want to do currency programming and you want to pass data between different tasks, then thread safe queues are quite a simple effective way to orchestrate the demand complexity in my experience.
After all that's how parallel processing is managed in the Post Office.
Re:How to choose the right Python concurrency API
by uniqueuid on Wednesday August 10, 2022 @05:40AM
(#32409591)
To be honest, your best cheap shot for making pipelines fast is to get all your data into RAM and then run your models. Ingestion i/o has lots of surprising bottlenecks, from small file i/o to NFS to decoding e.g. of image/video frames.
If you can afford it, create a standardized representation for your data and keep it in memory as much as possible. If that's not feasible, write the parsed representation into uncompressed tar files and load these on batch start.
Would you then separate the logic in such a way that you can use both multi-processing and multi-threading/co-routine. Is this possible when using async? I have limited experience but it feels like the moment you introduce async everything has to be async in the code.