650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

Re:650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. D... by esafak on Thursday November 13, 2025 @07:42PM (#45922556)

I used pySpark some time ago when it was introduced to my company at the time and I realized that it was slow when you used python libraries in the UDFs rather than pySpark's own functions.

[ Parent | on HN ]

Re:650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. D... by rmnclmnt on Friday November 14, 2025 @08:10AM (#45926384)

Yes using Python UDFs within Spark pipelines are a hog! That’s because the entire Python context is serialized with cloudpickle and sent over the wire to the executor nodes! (It can represent a few GB of serialized data depending on the UDF and driver process Python context)

[ Parent | on HN ]

Re:650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. D... by jellyfishbeaver on Sunday November 16, 2025 @09:57AM (#45945564)

We actually baked a rule to catch UDF usage into our Python linter. Almost always, a UDF can be refactored to use only native PySpark functions.

[ Parent | on HN ]