News for hackers. Stuff that matters.
Re:Blaze: Fast query execution engine for Apache Spark by whinvik on Friday October 20, 2023 @05:34PM (#37961814)
Any comparisons with Databricks Spark. When we started experimenting with Spark, we initially used AWS EMR. But then the same code was way faster on Databricks than it was on EMR, which resulted in us ditching EMR.
Re:Blaze: Fast query execution engine for Apache Spark by juliangamble on Friday October 20, 2023 @06:06PM (#37962035)
It would be great to have a comparison to Dataframes and RDDs as well.
Re:Blaze: Fast query execution engine for Apache Spark by bllchmbrs on Friday October 20, 2023 @07:31PM (#37962657)
DataFrames are just SQL. There will be no performance difference.

RDDs will be worse, so it shouldn't matter. No vectorization, no column processing, lots of serialization and de-serialization. They're basically always slower than DataFrames barring some strange use case.

Re:Blaze: Fast query execution engine for Apache Spark by esafak on Friday October 20, 2023 @06:06PM (#37962036)
Got numbers?
Re:Blaze: Fast query execution engine for Apache Spark by AdamProut on Friday October 20, 2023 @06:15PM (#37962098)
Databricks has kept their Photon[1][2] query engine for Spark closed sourced thus far. Unless EMR has made equivalent changes to the Spark runtime they use Databricks should be much faster. Photon brings the standard vectorized execution techniques used in SQL data warehouses for many years to Spark.

[1] https://docs.databricks.com/en/clusters/photon.html [2] https://dl.acm.org/doi/10.1145/3514221.3526054

Re:Blaze: Fast query execution engine for Apache Spark by whinvik on Saturday October 21, 2023 @07:20AM (#37965879)
I am a bit hazy about the exact details of how we did it since its been some time, but we definitely did not use Photon as it was too expensive.

One of the issues was that we started experimenting with Delta Tables and EMR was horrible in leveraging that.

Re:Blaze: Fast query execution engine for Apache Spark by elromulous on Friday October 20, 2023 @05:38PM (#37961855)
For a split second, I thought bazel[0] finally got externally renamed to its true name.

[0] https://en.m.wikipedia.org/wiki/Bazel_(software)

Re:Blaze: Fast query execution engine for Apache Spark by kristjansson on Friday October 20, 2023 @06:47PM (#37962333)
Unfortunate name overlap with an under-loved PyData project: https://blaze.pydata.org
Re:Blaze: Fast query execution engine for Apache Spark by tomrod on Friday October 20, 2023 @09:17PM (#37963251)
Same. I've always liked the blaze project.
Re:Blaze: Fast query execution engine for Apache Spark by tjpnz on Friday October 20, 2023 @11:01PM (#37963726)
And Google's version of Bazel.
Re:Blaze: Fast query execution engine for Apache Spark by laurentlb on Saturday October 21, 2023 @02:41AM (#37964630)
The public version was renamed Bazel because of name conflicts.
Re:Blaze: Fast query execution engine for Apache Spark by zX41ZdbW on Friday October 20, 2023 @09:26PM (#37963285)
Interesting, looks like it is just DataFusion engine for Spark. There is a similar project: https://github.com/oap-project/gluten - it brings ClickHouse as an engine to Spark.
Re:Blaze: Fast query execution engine for Apache Spark by zhangyt26 on Monday October 23, 2023 @08:38AM (#37984798)
Photon, velox, and now this. Why would people use spark in the first place other than for legacy application reasons?