News for hackers. Stuff that matters.

A comment on DuckDB 0.7.0. Parent comment

Re:DuckDB 0.7.0 by isoprophlex on Monday February 13, 2023 @05:10AM (#34771877)
Quick, in-core data transformation. If you want to transform some data right now, one option is writing pyspark and running that on a spark cluster. But noone really has big big data, there are relatively few cases where you have multi TB datasets, warranting the complexities of running the analyics in a distributed way.

DuckDB lets you process all that locally. It's the OLAP equivalent to SQLite's OLTP.

If I wasn't so beholden to the vagaries and inefficiencies of C-level endorsed enterprise software, I'd immediately be trying this out for data transformations/pipelines. I think that one big box (200+ gb ram, couple of cores and fat IO/network) runs circles around an entire spark cluster.

[ Parent | on HN ]
Re:DuckDB 0.7.0 by smt88 on Monday February 13, 2023 @12:23PM (#34776707)
Interesting. I need to think about this one a little bit. Thank you.

Is there a reason "in-core" is a specific requirement here?

[ Parent | on HN ]
Re:DuckDB 0.7.0 by isoprophlex on Monday February 13, 2023 @01:30PM (#34777920)
Not really, and duckdb doesn't need to hold everything in RAM as i recalll. But it's fast, far faster than several read-process-write steps can be, especially when coordinated over multiple machines

(By the way, maybe I was vague, using overloaded terminology. To be precise with 'in-core' i meant that the solution to an analytic query is held completely in memory, not that it's restricted to using one cpu thread.)

[ Parent | on HN ]