We ran with 0 hosts so in between requests, we had a lot of things failing or getting killed. The pattern of firing up a background thread from a request, returning immediately, and processing in the background simply did not work for us. The same logic worked fine elsewhere so it really was the environment. You can't really use it like a stateful environment.

In some cases we saw no evidence of the job even starting because the whole thing died right after the request returned. In other cases it would start but never finish. The actual import jobs weren't that big even. It was just super flaky.

The whole problem went away as soon as we gave it a dedicated vm. All I did was run the same container and then trigger the same request via a curl command sshed into the vm. Ugly but it got the job done.