650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

Re:650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. D... by co0lster on Thursday November 13, 2025 @07:16PM (#45922400)

650GB relates to size of parquet files which are compressed in reality it’s way more.

32 GB of parquet cannot fit in 32GB of RAM

[ Parent | on HN ]

Re:650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. D... by barrkel on Thursday November 13, 2025 @08:37PM (#45922884)

This would speed things up since it looks like the bottleneck here is I/O.

[ Parent | on HN ]

Re:650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. D... by m00x on Friday November 14, 2025 @12:23AM (#45924123)

You don't need it to if you just need specific columns. This is the advantage of columnar storage.

[ Parent | on HN ]