News for hackers. Stuff that matters.
Re:Distributed coroutines with a native Python extension and... by achille-roussel on Sunday March 24, 2024 @12:30PM (#39808198)
Dispatch takes a different approach with different trade-offs, but you're right that capturing the local program scope brings interesting challenges.

Serializing file handles doesn't work, but in our experience, programs rarely run into constructs where this becomes a problem, and when it happens, there are mitigation measures that are usually easy to implement (small restructure of the program, capturing resource metadata to reconstruct them later, etc...).

We have a few features on the roadmap to help mitigate the security implications as well, including allowing users to store their program state in a S3 bucket that they own. Our scheduler can operate with only metadata about the program, so splitting the two can be an effective model to mitigate those risks.

Re:Distributed coroutines with a native Python extension and... by richieartoul on Sunday March 24, 2024 @01:33PM (#39808723)
Storing the data in the customer's S3 bucket is a really good idea IMO. Means they just have to trust you to run the coordination reliably, but not necessarily to keep their data secure which is a much easier bar to clear.
Re:Distributed coroutines with a native Python extension and... by traverseda on Friday March 29, 2024 @11:32AM (#39865300)
What about combining it with rpyc for network transparent object proxies? For resources that can't be picked.