Thank you for taking the time to respond. That is only part of the problem. I am quite aware of that there are quite a number of external issues that can affect a job. Data extraction is something I have been working in for more than a decade.

I have a few other pet peeves with Celery. I run scheduled tasks using CeleryBeat but those tasks cannot be tracked from flower. Signals don't work. I also would like something language agnostic so I can write memory/processing intensive tasks in something more performant (Go or Rust).