You seem to be assuming that preemption is always needed or wanted. This is not always the case. The lack of preemption actually simplifies a number of factors that simply aren't relevant in async scenarios in a single program.

async/await patterns are actually the perfect solution for capturing the minimal context necessary for resumption after the async operation completes.

It also have the benefit of making concurrent access much easier to reason about without having to resort to message passing or immutability, yet locks are rarely necessary.

As usual, it's a trade off.

And on a bad day, I can do "the bad thing" to myself.

One way to help mitigate this is good tooling. A warning that a piece of async computation took longer than expected can be very useful if you expect everything to finish in a few tens of milliseconds.

As an example of cooperative multitasking with dataflow, consider single-process nginx. Events are delivered using epoll; nginx has an event loop that does work on the next task that has data ready (or a timer event.) This is called the Reactor Pattern in the article.

Cooperative multitasking has much of the semantics (and thus semblance) of preemptive multitasking while also having the allure of not needing the harder (either with respect to understanding or actual usage) semantics of locking, the mental accounting related to thread safe algorithms, etc.

In other words, you get the semantics of concurrency (e.g. tasks and working queues) without the overhead of synchronizing. That sounds very alluring, but in practice I would like to not delve into specialized semantics AT ALL unless I'm expecting some speed/scalability gains, where the bulk of it comes from actual parallel execution, either in the form of different cores or different nodes. And then you need in any case to use the same semantics concurrent (i.e. cooperative) frameworks try to avoid, e.g. in the form of synchronizing between processes but with much less tooling because this is a second-tier scenario in most cooperative multitasking frameworks I've seen.

In other words, you don't get the speedups that are worth the mental effort of using the semantics in the first place. The only exception I've seen is a code base properly utilizing their I/O "breaks" for cooperation and indeed I see cooperative multitasking frameworks often touting I/O intensive applications as an ideal candidate. But, please note that:

1. Programmers usually overestimate how much I/O (or any resource usage) blocks. The codebase I was referring to was an actual filesystem.

2. Behind toy programs you still need to do a lot of thread-safe type accounting. You're not susceptible to cache-coherency issues, but that's about it.

Phew, didn't think I have so much to say about it.

The delay in the queue simulates a polling interval to the external system and when a successful response is returned from the external system the Dynamo record is updated with a status telling the Lambda subscribing to the event stream that "the loop" is done.

This setup has to run at fairly significant peak loads and we're relying on Amazon to provision resources reliably enough to pull this off. None of us know if this will work but in terms of abstracting away the complexities of async/concurrent processing it's a novel approach.

It’s actually similar than an in-program model where you just asynchronously queue void functions in threadpools without knowing how many actually run.

The fact that you are only running on top of managed services and that AWS can scale really well might mitigate the problem. And if you have to process all events and can not fail early or move backpressure on the initial caller there might not really be a better solution anyway. But you should be aware about it.