I think they mean that they may need thousands of HPC cores for a few hundred milliseconds, and then zero cores right after; and (for some reason) absolutely cannot just do the work serially instead of in parallel.

Yeah, I also interpret it that way, but I cannot understand what kind of system requires such a scheme. Even if the state space is non-ergodic, you should be able to get by with running your sampling on a dozen larger systems in some sense. Or just run on a workstation for a few months to precompute and tabulate the result for all possible inputs.

I realize this was a week ago, but the parent comment is basically the case. The "some reason" is the expected time from request->response (which includes aggregation/interpretation of results) is 1-2 seconds with a sample size of ~50,000 for a pretty involved model that needs to be run on demand/in "real time" with no good, predictable time window.

We can definitely get by running the sampling on a dozen larger systems, but those systems still cost more than (one thousand Lambda executions * a few hundred times a week max).

Pre-computing is an interesting idea, but would certainly be an epic amount of work. The input set is also quite large (compiled motorsports data based on current state of an event).

I'm also totally willing to admit I'm probably in over my head in designing such a system, but I did try a few different ways of architecting this and tweaking the parallelization in the source and this was the best cross of time/money I could get! It's definitely been a learning process for me; although I've got some years experience with scaling out, the system and requirements were much different than I've been exposed to. I'm sure this first take will be changed or at least refined once it's been in use for awhile.