Re:Snowflake’s response to Databricks’ TPC-DS post
by hiyer on Saturday November 13, 2021 @02:30AM
(#29208106)
I'm aware that Databricks is a PaaS service, but what Databricks runs under the hood is Spark (with a few proprietary extensions). So your jobs/queries do require some tuning just like with OS Spark.
Spark has had SQL engines (SparkSQL/Hive on Spark) for a long time. Photon is just a new, faster one. Photon tasks also run on Spark executors only, so it's not independent of Spark[1]. Also, while it's proprietary now, I wouldn't be surprised if Databricks open-sources it in the future, like they did with Delta Lake.
Spark has had SQL engines (SparkSQL/Hive on Spark) for a long time. Photon is just a new, faster one. Photon tasks also run on Spark executors only, so it's not independent of Spark[1]. Also, while it's proprietary now, I wouldn't be surprised if Databricks open-sources it in the future, like they did with Delta Lake.
1. https://databricks.com/blog/2021/06/17/announcing-photon-pub...