Otherwise when the logic gets complex I try to model it as independent threads each having a well-defined role (e.g. exclusive write access to SQLite) communicating with channels. Although I tend to code these things in Rust, the language who taught me that philosophy is Go.
Design-wise:
The IO multiplexing — event loop(s), the completion callbacks go to the "Processing".
Processing — usually beneficial to model as a processing graph (tree/pipeline being specific examples of a graph without feedback loops). It's a clean, low overhead abstraction, easy to visualise (and thus discuss with others), easy to change structurally or in the "aspect oriented" way.
Examples in JVM world: Netty pipelines, AKKA actors, streams, and graphs, in C++ world ASIO & its executors sub-framework.
Queues/Threads is still a sufficiently low level to comfortably operate on.
group -> make concurrent calls and wait -> sort
which would have been very straightforward, and probably faster too because you don't let any one step interrupt the other
The task at hand wasn’t easy at all, it required both:
1. Making numerous network requests (some endpoints support batching, so do that as much as possible) 2. The entire main loop needs to be as fast as reasonably possible
The only real way to satisfy both, given that serially executing the network activity would waste tons of time, was some fairly tricky concurrent logic. I ended up using a few queue structures to collect day before making batch network calls, and stitching together all of the results at the end (block on all work threads, collect, report, then start next iteration)
Every step along the way made sense, every addition of complexity felt justified. And the first person to come to me asking how the fuck it got so complicated is frustrated, so my knee jerk reaction is to get frustrated back saying well how else could we satisfy complex requirements with a complex solution.
We should always be challenging ourselves and others to keep solutions simple, but remember that the simplest solution may itself still be complex.