News for hackers. Stuff that matters.
Re:How to choose the right Python concurrency API by ansre on Wednesday August 10, 2022 @12:43PM (#32414913)
I'm curious if you've looked into https://docs.celeryq.dev/en/stable/index.html . I've used it for relatively large batch processing jobs on AWS ECS. When I was maintaining this system I had between 100-200 EC2 instances all running 1 celery worker per core (IIRC). It was relatively simple to maintain and scaled very well.

Celery is great for scaling out jobs horizontally, but I don't know if that's the actual problem you were facing with your current system. If your problem is with an individual job not processing/completing fast enough, then yeah maybe python isn't the right choice for you ¯\_(ツ)_/¯.

Re:How to choose the right Python concurrency API by jonstewart on Wednesday August 10, 2022 @09:59PM (#32420745)
Scaling horizontally is well and good (since my unit of input is large and I get a lot of them, I just use autoscaling groups in AWS to achieve much the same), but I don’t want to pay for 10 systems slowly doing what 1 could do with fast software. Especially, while 1 system has a hard ceiling, there’s a lot of headroom on 1 system when you achieve mechanical sympathy with the cache hierarchy, memory controller, and disk.