Apache Drill's Future

Re:Apache Drill's Future by yachtman on Thursday May 21, 2020 @12:35AM (#23254836)

Was SparkSQL ever intended to replace hive? My impression was that it was supposed to supplement spark for times it was convenient. I kind of suspected at one point they got caught up in the SQL hadoop race, but I always felt like it was best to do SQL elsewhere, and save spark for things that couldn't be easily expressed in SQL.

[ Parent | on HN ]

Re:Apache Drill's Future by artwr on Thursday May 21, 2020 @01:35AM (#23255145)

The original SparkSQL was pretty much modelled after the Hive flavour of SQL, down to the available udfs. The compatibility was never complete and has somewhat diverged again with respective releases of the frameworks, but for the most part, Hive was the big data framework to beat at the time (2015-2016), and not everyone wanted to write Scala.

I think that now, maintaining that compatibility is less of a need for Spark and Hive has introduced a lot of goodies in the meantime, so there might not be a need for the SQL flavors to be in lockstep anymore.

[ Parent | on HN ]

Re:Apache Drill's Future by jlj on Thursday May 21, 2020 @01:33AM (#23255132)

SQL can be used as a dataframe, or a hive temp view that can be called from other SQL. That gives flexibility to mix and match SQL and programmatic logic within the same spark app.

[ Parent | on HN ]