The type of computation you're modeling is limited by the async/dispatched nature of celery. Threads can communicate data structures to each other. With celery, that's awkward with the latency and async semantics.
[1] I've used rabbitmq to dispatch jobs over to celery in the past, here's a good overview of latency in that process. TLDR requests that hit the wire are on the order of milliseconds: https://www.rabbitmq.com/blog/2012/05/11/some-queuing-theory...
Celery is suitable for "send an email" or "rebuild a value and put in memcache" but not suitable for "when the OS finally returns the file handle I requested, could you read the file into memory please"
Totally different time scales and use-cases.
I built an angular button that would call a task and would show spinning in-progress and then show when the task completes. Works on development, didn't work on production. why why why ? well, no time to fiddle with celery so I moved on. over the years I've come to mistrust its complexity.
Yeah, I think nested/chained tasks are pointless. The point of celery is to offload the task to an asychronous worker (probably on another machine) so if you're already on the worker there's no point sending another task to the same worker--it doesn't achieve anything above doing the task inline.
> I don't use periodic tasks because they can pile up badly and block everything.
This sounds like you're either running the periodic task too frequently, or you need more workers. I've seen a few cases where people tried running a 20-minute task every second on a single worker--of course this is going to pile up. Periodic tasks are among the most predictable, so they're easiest to prevent from piling up: simply make sure that the following relation holds true:
w > t / p
w = number of workers
t = how long it takes for the task to run
p = the period (a task that runs ever 20 minutes has period 20 minutes)
The > should be greater than by an order of magnitude to be safe in case t increases.Also increasing workers isn't so simple. There is only so much memory on a machine, and I don't need or want to add machines. A more efficient loop was the better solution. The machine is now full of work all the time and stable in memory, CPU usage and db access.
This is no longer a problem now that I don't use periodic tasks. "periodic" doesn't describe the nature of the work to be done.
In any case the main problem I needed to solve is that I don't have single tasks that need to happen every p seconds. I have 40 tasks that have different min-max-acceptable-backlog-queue-length, different min-max time periods that they could run in. Different day and night cycle ranges. Some really need to go every 2 minutes, but only if there are emails to send. Some are things to do whenever there is spare time. Some image processing queues need to be done very quickly, for others it can chew on it all night when there are spare cycles.