Photon powered Engine (enhance Apache Spark 3.0’s performance by up to 20x)

Posted by techempower on Friday October 02, 2020 @09:01PM
from the techcommunity.microsoft.com dept.

https://techcommunity.microsoft.com/t5/azure-databricks/turbocharge-azure-databricks-with-photon-powered-delta-engine/ba-p/1694929

( 2 comments | 7 points | on HN )

Re:Photon powered Engine (enhance Apache Spark 3.0’s perform... by alphachloride on Friday October 02, 2020 @11:13PM (#24669207)

For just a moment I was pleasantly surprised that I would be reading about an interesting propulsion technology. Seeing as that current propulsion approaches rely on ejecting Fermion composites at high relative velocities to the engine.

I am not pleasantly surprised anymore.

[ Parent | on HN ]

Re:Photon powered Engine (enhance Apache Spark 3.0’s perform... by legg0myegg0 on Saturday October 03, 2020 @01:58AM (#24669902)

DuxkDB's query engine is inspired by the same paper! You'll be surprised what you can process on a single node with DuckDB - it takes 33 Spark nodes to match performance! That still puts it ahead of Photon, and it is open source.

[ Parent | on HN ]