News for hackers. Stuff that matters.
Re:Amazon's Exabyte-Scale Migration from Apache Spark to Ray... by robertnishihara on Wednesday July 31, 2024 @07:12PM (#41124520)
I'm glad you find it exciting!

Our intention from the start was for Ray to be general purpose. And the core Ray APIs are quite general (basically just scheduling a Python function somewhere in a cluster or instantiating a Python class as a process somewhere in the cluster).

We had AI use cases in mind from the start, since we were grad students in AI. But the generality has really been important since AI workloads encompass a huge variety of computational patterns (allreduce style communication patterns on GPUs for training, embarrassingly parallel data processing workloads on spot instances, and so on).

Re:Amazon's Exabyte-Scale Migration from Apache Spark to Ray... by dekhn on Wednesday July 31, 2024 @07:31PM (#41124656)
Oh, I know all that, I used to work at Google and give lots of money to the various groups associated with Ion Stoica's groups at Berkeley to help stimulate more open source alternatives to Borg/MapReduce/Flume/TensorFlow. Keep up the good work.
Re:Amazon's Exabyte-Scale Migration from Apache Spark to Ray... by refset on Thursday August 01, 2024 @08:26AM (#41128433)
Is there anybody trying to build a SQL database on Ray yet? Asking for a friend.