News for hackers. Stuff that matters.
Re:DataFusion Comet: Apache Spark Accelerator by orthoxerox on Friday May 31, 2024 @03:25PM (#40539256)
Because it's a drop-in replacement that lets you (theoretically) spend O(1) development effort on speeding up your Spark jobs instead of O(N).

I say theoretically, because I have no idea how Comet works with the memory limits on Spark executors. If you have to rebalance the memory between regular memory and memory overhead or provision some off-heap memory for Comet, then the migration won't be so simple.

Re:DataFusion Comet: Apache Spark Accelerator by nicornk on Saturday June 01, 2024 @12:39AM (#40542923)
and this is unfortunately the case for Velox.
Re:DataFusion Comet: Apache Spark Accelerator by _zoltan_ on Saturday June 01, 2024 @01:20PM (#40547488)
care to elaborate?