News for hackers. Stuff that matters.
Re:A sharded DuckDB on 63 nodes runs 1T row aggregation chal... by vysakh0 on Friday October 24, 2025 @12:34PM (#45696289)
Duckdb is an excellent OLAP db, I have had customers who had s3 data lake of parquet and use databricks or other expensive tool, when they could easily use duckdb.. Given we have cursor/claude code, it is not that hard for lot of use cases, I think the lack of documentation on how duckdb functions -- in terms of how it loads these files etc are some of the reasons companies are not even trying to adopt duckdb. I think blogs like this is a great testament for duckdb's performance!
Re:A sharded DuckDB on 63 nodes runs 1T row aggregation chal... by lopatin on Friday October 24, 2025 @01:08PM (#45696700)
DuckDB is great but it’s barely OLAP right? A key part of OLAP is “online”. Since the writer process blocks any other processes from doing reads, calling it OLAP is a stretch I think.
Re:A sharded DuckDB on 63 nodes runs 1T row aggregation chal... by ansgri on Friday October 24, 2025 @02:17PM (#45697514)
Isn't the Online part here about getting results immediately after query, as opposed to overnight batch reports? So if you don't completely overwhelm DuckDB with writes, it still qualifies. The quality you're describing is something like "realtime analytics", and is a whole another category: Clickhouse doesn't qualify (batching updates, merging etc. — but it's clearly OLAP), Druid does.
Re:A sharded DuckDB on 63 nodes runs 1T row aggregation chal... by lopatin on Friday October 24, 2025 @02:47PM (#45697833)
Huh yeah looks like I was totally wrong about what online meant. So yeah DuckDB is OLAP. Not that anyone was asking me in the first place. Carry on :)
Re:A sharded DuckDB on 63 nodes runs 1T row aggregation chal... by sdairs on Friday October 24, 2025 @02:49PM (#45697848)
ClickHouse is the market leader in real-time analytics so it's an interesting take that you don't think it qualifies.
Re:A sharded DuckDB on 63 nodes runs 1T row aggregation chal... by ansgri on Friday October 24, 2025 @05:42PM (#45699399)
For certain definition of realtime, certainly (as would any system with bounded ingestion latency), but it’s not low-latency streaming realtime. Tens of seconds or more can pass before new data becomes visible in queries in normal operation. There’s batching, there’s merging, and its overall architecture prioritizes throughput over latency.
Re:A sharded DuckDB on 63 nodes runs 1T row aggregation chal... by mrtimo on Friday October 24, 2025 @12:44PM (#45696395)
I have experience with duckDB but not databricks... from the perspective of a company, is a tool like databricks more "secure" than duckdb? If my company adopts duckdb as a datalake, how do we secure it?
Re:A sharded DuckDB on 63 nodes runs 1T row aggregation chal... by rapatel0 on Friday October 24, 2025 @01:12PM (#45696754)
Duckdb can run as a local instance that points to parquet files in a n s3 bucket. So your "auth" can live on the layer that gives permissions to access that bucket.
Re:A sharded DuckDB on 63 nodes runs 1T row aggregation chal... by adammarples on Friday October 24, 2025 @02:59PM (#45697974)
I have been playing today with ducklake, and I have to confess I don't quite get what it does that duckdb doesn't already do, if duckdb can just run on top of parquet files quite happily without this extension...
Re:A sharded DuckDB on 63 nodes runs 1T row aggregation chal... by RobinL on Friday October 24, 2025 @04:01PM (#45698521)
It's main purpose is to solve the problem of upserts to a data lake, because upsert operations to file based data storage are a real pain.