Re:Show HN: Pyper – Concurrent Python Made Simple
by JackC on Wednesday January 15, 2025 @10:21AM
(#42711848)
Cool, I was just looking for something like this!
It's surprisingly annoying in built-in python to do something like this. The most recent thing I was trying to do was:
- load URLs from a file
- hand them out to one subprocess per cpu
- download them concurrently in threads or async within each subprocess
- pull the results back into a single process for formatting and storing
Getting this to work and handle queues, ctrl-c, exceptions etc. is just a whole mess involving python builtins created at different times with different interfaces; I hacked until I kind of got it working, but didn't love it. Bundling it all in a single tested package would be great.
It's surprisingly annoying in built-in python to do something like this. The most recent thing I was trying to do was:
- load URLs from a file - hand them out to one subprocess per cpu - download them concurrently in threads or async within each subprocess - pull the results back into a single process for formatting and storing
Getting this to work and handle queues, ctrl-c, exceptions etc. is just a whole mess involving python builtins created at different times with different interfaces; I hacked until I kind of got it working, but didn't love it. Bundling it all in a single tested package would be great.