Re:How to choose the right Python concurrency API
by jonstewart on Wednesday August 10, 2022 @09:59PM
(#32420745)
Scaling horizontally is well and good (since my unit of input is large and I get a lot of them, I just use autoscaling groups in AWS to achieve much the same), but I don’t want to pay for 10 systems slowly doing what 1 could do with fast software. Especially, while 1 system has a hard ceiling, there’s a lot of headroom on 1 system when you achieve mechanical sympathy with the cache hierarchy, memory controller, and disk.
Celery is great for scaling out jobs horizontally, but I don't know if that's the actual problem you were facing with your current system. If your problem is with an individual job not processing/completing fast enough, then yeah maybe python isn't the right choice for you ¯\_(ツ)_/¯.