Re:Apache Arrow 3.0
by yeshengm on Wednesday February 03, 2021 @06:20PM
(#26020563)
Are these execution engines internally using Arrow columnar format or are they just exposing Arrow as a client wire format? AFAIK Spark and Presto does not use Arrow as execution columnar format, but just data sources/sinks.
[ Parent
| on HN ]
Re:Apache Arrow 3.0
by groceryheist on Wednesday February 03, 2021 @08:15PM
(#26021407)
You can configure Spark to use arrow for passing data between Java and Python via spark.sql.execution.arrow.pyspark.enabled but yes, Spark uses Java datatypes internally.