News for hackers. Stuff that matters.

A comment on Summing columns in remote Parquet files using DuckDB. Parent comment

Re:Summing columns in remote Parquet files using DuckDB by simonw on Tuesday November 14, 2023 @08:29PM (#38272246)
Do you know how much data it had to download to run that query? Did it pull all 8GB?
[ Parent | on HN ]
Re:Summing columns in remote Parquet files using DuckDB by zX41ZdbW on Wednesday November 15, 2023 @04:50AM (#38274994)
The same query takes only 1.4 seconds on my server, so I assume that the query does not read all 8 GB.
[ Parent | on HN ]
Re:Summing columns in remote Parquet files using DuckDB by richraposa on Wednesday November 15, 2023 @09:58AM (#38277265)
ClickHouse just reads the values from the one column being summed: https://www.markhneedham.com/blog/2023/11/15/clickhouse-summ...
[ Parent | on HN ]
Re:Summing columns in remote Parquet files using DuckDB by hipadev23 on Thursday November 16, 2023 @08:50PM (#38298243)
I ran a network monitor while running that query, it pulled down ~290MB
[ Parent | on HN ]
Re:Summing columns in remote Parquet files using DuckDB by simonw on Thursday November 16, 2023 @08:59PM (#38298339)
Almost exactly the same as DuckDB then - I measured ~287MB.
[ Parent | on HN ]
Re:Summing columns in remote Parquet files using DuckDB by riku_iki on Thursday November 16, 2023 @09:01PM (#38298366)
Probably compression
[ Parent | on HN ]
Re:Summing columns in remote Parquet files using DuckDB by daveguy on Friday November 17, 2023 @11:16AM (#38305582)
If that's raw network transfer, it's probably just a difference in headers or MTU size. Larger MTU -> fewer headers required. Maybe a difference in network configuration that requires more or less data in the header.
[ Parent | on HN ]
Re:Summing columns in remote Parquet files using DuckDB by qxip on Sunday November 19, 2023 @05:36AM (#38331306)
I ran the same queries and got similar results but the bandwidth utilization I measured was significantly different. On the same fly.io instance with 1vCPU/256MB both queries completed successfully but ClickHouse/chdb reached 10MB/s (max) and logically completed the count faster, while DuckDB only peaked at around 2.5MB/s.

This might be due to the tiny resources but I like rock bottom measurements. Did anyone else notice a similar bandwidth utilization gap?

[ Parent | on HN ]