News for hackers. Stuff that matters.
Re:DuckDB over Pandas/Polars by wenc on Tuesday November 05, 2024 @11:01PM (#42057127)
Also, I learned that Hive-partitioned Parquet on S3 is much slower than on disk.

S3 is high latency unless you use for S3 Express Zones (the low latency version).

We used EFS (not EBS) and it was much faster.

Re:DuckDB over Pandas/Polars by ramraj07 on Tuesday November 05, 2024 @11:10PM (#42057156)
Test out the nvme drives though. It’s blazing.