Should You Ditch Spark for DuckDB or Polars?

Re:Should You Ditch Spark for DuckDB or Polars? by ritchie46 on Monday December 16, 2024 @12:58AM (#42428335)

I do think code should be shared when you are benchmarking. He could be using Polars' eager API for instance, which would not be apples to apples.

[ Parent | on HN ]

Re:Should You Ditch Spark for DuckDB or Polars? by mwc360 on Monday December 16, 2024 @01:18AM (#42428432)

Hi - Miles Cole here… I used lazy APIs where available. I.e. everything up to write_delta() is lazy in the Polars (Mod) variant.

Yeah I was debating whether to share all of the source code. I may share a portion of it soon.

[ Parent | on HN ]

Re:Should You Ditch Spark for DuckDB or Polars? by ritchie46 on Monday December 16, 2024 @02:54AM (#42428896)

Great! A small correction on your post. Polars does have SQL suppor. It isn't the main usecase, so it isn't as good as that of Spark and DuckDB, but it does exist and is being improved on.

[ Parent | on HN ]

Re:Should You Ditch Spark for DuckDB or Polars? by mwc360 on Monday December 16, 2024 @12:32PM (#42433182)

Ritchie - thx for graciously correcting some things I got wrong, will get it corrected!

[ Parent | on HN ]