News for hackers. Stuff that matters.

A comment on Should You Ditch Spark for DuckDB or Polars?. Parent comment

Re:Should You Ditch Spark for DuckDB or Polars? by mwc360 on Monday December 16, 2024 @01:32AM (#42428504)
Author here: that’s exactly what I was trying to communicate but you said it better :)
[ Parent | on HN ]
Re:Should You Ditch Spark for DuckDB or Polars? by cmdlineluser on Monday December 16, 2024 @11:26AM (#42432499)
There is a Spark API[1] being built using their Relational API[2].

Progress is being tracked on Github Discussions[3].

[1]: https://duckdb.org/docs/api/python/spark_api.html

[2]: https://duckdb.org/docs/api/python/relational_api.html

[3]: https://github.com/duckdb/duckdb/discussions/14525

[ Parent | on HN ]
Re:Should You Ditch Spark for DuckDB or Polars? by mwc360 on Monday December 16, 2024 @01:19PM (#42433684)
Very cool! This seems like fantastic functionality and would make it super easy to migrate small Spark workloads to DuckDB :)
[ Parent | on HN ]