Summing columns in remote Parquet files using DuckDB

Re:Summing columns in remote Parquet files using DuckDB by richraposa on Tuesday November 14, 2023 @06:16PM (#38271190)

Nice. I did the same thing on ClickHouse - took 11 seconds:

    SELECT sum(size)
    FROM url('https://huggingface.co/datasets/vivym/midjourney-messages/resolve/main/data/0000{01..55}.parquet')
    
    Query id: 9d145763-0754-4aa2-bb7d-f6917690f704
    
    ┌───────sum(size)─┐
    │ 159344011148016 │
    └─────────────────┘
    
    1 row in set. Elapsed: 11.615 sec. Processed 54.08 million rows, 8.50 GB (4.66 million rows/s., 731.83 MB/s.)
    Peak memory usage: 458.88 KiB.

[ Parent | on HN ]

Re:Summing columns in remote Parquet files using DuckDB by simonw on Tuesday November 14, 2023 @08:29PM (#38272246)

Do you know how much data it had to download to run that query? Did it pull all 8GB?

[ Parent | on HN ]

Re:Summing columns in remote Parquet files using DuckDB by zX41ZdbW on Wednesday November 15, 2023 @04:50AM (#38274994)

The same query takes only 1.4 seconds on my server, so I assume that the query does not read all 8 GB.

[ Parent | on HN ]

Re:Summing columns in remote Parquet files using DuckDB by richraposa on Wednesday November 15, 2023 @09:58AM (#38277265)

ClickHouse just reads the values from the one column being summed: https://www.markhneedham.com/blog/2023/11/15/clickhouse-summ...

[ Parent | on HN ]

Re:Summing columns in remote Parquet files using DuckDB by hipadev23 on Thursday November 16, 2023 @08:50PM (#38298243)

I ran a network monitor while running that query, it pulled down ~290MB

[ Parent | on HN ]

Re:Summing columns in remote Parquet files using DuckDB by simonw on Thursday November 16, 2023 @08:59PM (#38298339)

Almost exactly the same as DuckDB then - I measured ~287MB.

[ Parent | on HN ]

Re:Summing columns in remote Parquet files using DuckDB by riku_iki on Thursday November 16, 2023 @09:01PM (#38298366)

Probably compression

[ Parent | on HN ]

Re:Summing columns in remote Parquet files using DuckDB by daveguy on Friday November 17, 2023 @11:16AM (#38305582)

If that's raw network transfer, it's probably just a difference in headers or MTU size. Larger MTU -> fewer headers required. Maybe a difference in network configuration that requires more or less data in the header.

[ Parent | on HN ]

Re:Summing columns in remote Parquet files using DuckDB by qxip on Sunday November 19, 2023 @05:36AM (#38331306)

I ran the same queries and got similar results but the bandwidth utilization I measured was significantly different. On the same fly.io instance with 1vCPU/256MB both queries completed successfully but ClickHouse/chdb reached 10MB/s (max) and logically completed the count faster, while DuckDB only peaked at around 2.5MB/s.

This might be due to the tiny resources but I like rock bottom measurements. Did anyone else notice a similar bandwidth utilization gap?

[ Parent | on HN ]

Re:Summing columns in remote Parquet files using DuckDB by zxexz on Thursday November 16, 2023 @09:19PM (#38298515)

I love and use both clickhouse and duckdb all the time, but it's become almost comical how senior clickhouse leadership comment on any duckdb-related HN thread like clockwork. And only very rarely bring up their affiliation :)

[ Parent | on HN ]

Re:Summing columns in remote Parquet files using DuckDB by chatmasta on Thursday November 16, 2023 @09:37PM (#38298667)

You mean this username composed of a random assortment of Latin characters isn't an unaffiliated, independent commentator?!

[ Parent | on HN ]

Re:Summing columns in remote Parquet files using DuckDB by richraposa on Thursday November 16, 2023 @10:06PM (#38298951)

...says the completely anonymous internet guy. I'm laughing as much as you are.

I wasn't trying to imply anything negative about DuckDB with my post - was just sharing how ClickHouse does the same thing. FWIW: the blog author added my query to his blog, so my non-combative comment was politely received.

[ Parent | on HN ]

Re:Summing columns in remote Parquet files using DuckDB by swasheck on Thursday November 16, 2023 @11:02PM (#38299398)

but what the person said is true: seems like clickhouse comments descend upon every recent duckdb post as if it’s some sort of competition or born out of inferiority complex.

clickhouse is really cool tech. duckdb is really cool tech. i grow weary of the CH infiltration to the point where it’s working against the intent to make me love CH

[ Parent | on HN ]

Re:Summing columns in remote Parquet files using DuckDB by datadeft on Friday November 17, 2023 @06:03AM (#38301912)

Is it so annoying? I am happy that we have at least two projects that can query big datasets with very reasonable performance.

[ Parent | on HN ]

Re:Summing columns in remote Parquet files using DuckDB by cmdlineluser on Friday November 17, 2023 @06:21AM (#38302021)

It's "interesting". https://news.ycombinator.com/item?id=34772367

[ Parent | on HN ]

Re:Summing columns in remote Parquet files using DuckDB by swasheck on Friday November 17, 2023 @09:26AM (#38304018)

yes it is quite annoying. i would love to see a CH dedicated post to extol its virtues and not a piggyback “us too” comment on a duckdb post.

[ Parent | on HN ]

Re:Summing columns in remote Parquet files using DuckDB by zxexz on Friday November 17, 2023 @11:38AM (#38305928)

Hey man - not trying to get on the bad side of one of my favourite databases! Making an observation, one that others have made as well.

[ Parent | on HN ]