News for hackers. Stuff that matters.

A comment on 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark.

Re:650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. D... by dogman123 on Thursday November 13, 2025 @09:27PM (#45923189)
One thing that I never really see mentioned in these types of articles is that a lot of DuckDB’s functionality does not work if you need to spill to disk. iirc, percentiles/quartiles (among other aggregate functions) caused DuckDB to crash out when it spilled to disk.
[ Parent | on HN ]
Re:650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. D... by jtbaker on Thursday November 13, 2025 @09:40PM (#45923261)
I’m pretty sure I’ve done this and not had any issues. Can you share a minimum reproducible example?
[ Parent | on HN ]