News for hackers. Stuff that matters.
Re:Why DuckDB is my first choice for data processing by wood_spirit on Friday January 16, 2026 @03:17PM (#46651621)
I’m also a duckdb convert. All my notebooks have moved from Pandas and polars to Duckdb. It is faster to write and faster to read (after you return to a notebook after time away) and often faster to run. Certainly not slower to run.

My current habit is to suck down big datasets to parquet shards and then just query them with a wildcard in duckdb. I move to bigquery when doing true “big data” but a few GB of extract from BQ to a notebook VM disk and duckdb is super ergonomic and performant most of the time.

It’s the sql that I like. Being a veteran of when the world went mad for nosql it is just so nice to experience the revenge of sql.

Re:Why DuckDB is my first choice for data processing by theLiminator on Friday January 16, 2026 @03:25PM (#46651730)
I personally find polars easier to read/write than sql. Especially when you start doing UDFs with numpy/et. al. I think for me, duckdb's clear edge is the cli experience.

> It is faster to write and faster to read

At least on clickbench, polars and duckdb are roughly comparable (with polars edging out duckdb).

Re:Why DuckDB is my first choice for data processing by erikcw on Friday January 16, 2026 @09:49PM (#46654802)
I use them both depending on which feels more natural for the task, often within the same project. The interop is easy and very high performance thanks to Apache Arrow: `df = duckdb.sql(sql).pl()` and `result = duckdb.sql("SELECT * FROM df")`.
Re:Why DuckDB is my first choice for data processing by exagolo on Monday January 19, 2026 @05:23AM (#46677247)
For the bigger tasks, Exasol might also be a very neat option for you. We have a free personal edition that can scale regarding data volumes, #servers (MPP architecture) and complex workloads.

Recently, we have also compared ourselves against DuckDB and were 4 times faster even on a single node. We are in-memory optimized, but data doesn't need to fit in the RAM.

Disclaimer: I'm CTO@Exasol