Re:Amazon's Exabyte-Scale Migration from Apache Spark to Ray...
by theLiminator on Wednesday July 31, 2024 @07:40PM
(#41124707)
Curious if you know how well Ray works with multithreaded python libraries? For example, when using jax with ray, I have to ensure the import ordering imports ray first, as forking a threaded process leads to deadlocks in Python. Do you know how to ensure that ray handles forking the python interpreter correctly?
[ Parent
| on HN ]
Re:Amazon's Exabyte-Scale Migration from Apache Spark to Ray...
by robertnishihara on Wednesday July 31, 2024 @08:21PM
(#41124965)
Multi-threaded libraries (e.g., numpy and PyTorch on CPUs come to mind) are well supported. In scenarios where many processes are each running heavily multi-threaded computations, it can help to pin specific processes to specific cores (e.g., using tools like psutil) to avoid contention.
The scenario where a Ray task forks is probably not very well supported. You can certainly start a subprocess from within a Ray task, but I think forking could easily cause issues.
You can definitely use Ray + Jax, but you probably need to avoid forking a process within a Ray worker.