Summing columns in remote Parquet files using DuckDB

Re:Summing columns in remote Parquet files using DuckDB by simonw on Thursday November 16, 2023 @08:59PM (#38298339)

Almost exactly the same as DuckDB then - I measured ~287MB.

[ Parent | on HN ]

Re:Summing columns in remote Parquet files using DuckDB by riku_iki on Thursday November 16, 2023 @09:01PM (#38298366)

Probably compression

[ Parent | on HN ]

Re:Summing columns in remote Parquet files using DuckDB by daveguy on Friday November 17, 2023 @11:16AM (#38305582)

If that's raw network transfer, it's probably just a difference in headers or MTU size. Larger MTU -> fewer headers required. Maybe a difference in network configuration that requires more or less data in the header.

[ Parent | on HN ]

Re:Summing columns in remote Parquet files using DuckDB by qxip on Sunday November 19, 2023 @05:36AM (#38331306)

I ran the same queries and got similar results but the bandwidth utilization I measured was significantly different. On the same fly.io instance with 1vCPU/256MB both queries completed successfully but ClickHouse/chdb reached 10MB/s (max) and logically completed the count faster, while DuckDB only peaked at around 2.5MB/s.

This might be due to the tiny resources but I like rock bottom measurements. Did anyone else notice a similar bandwidth utilization gap?

[ Parent | on HN ]