Re:Distributed coroutines with a native Python extension and...
by richieartoul on Sunday March 24, 2024 @01:33PM
(#39808723)
Storing the data in the customer's S3 bucket is a really good idea IMO. Means they just have to trust you to run the coordination reliably, but not necessarily to keep their data secure which is a much easier bar to clear.
Re:Distributed coroutines with a native Python extension and...
by traverseda on Friday March 29, 2024 @11:32AM
(#39865300)
What about combining it with rpyc for network transparent object proxies? For resources that can't be picked.
Serializing file handles doesn't work, but in our experience, programs rarely run into constructs where this becomes a problem, and when it happens, there are mitigation measures that are usually easy to implement (small restructure of the program, capturing resource metadata to reconstruct them later, etc...).
We have a few features on the roadmap to help mitigate the security implications as well, including allowing users to store their program state in a S3 bucket that they own. Our scheduler can operate with only metadata about the program, so splitting the two can be an effective model to mitigate those risks.