Re:DuckDB – An embeddable SQL database like SQLite, but supp...
by legg0myegg0 on Sunday September 20, 2020 @12:00PM
(#24534721)
Before the latest optimization, and only using 1 core, vs. SQLite we were seeing 133x performance on a basic group by or join, and about 4x for a pretty complex query. It was roughly even to Pandas in performance, but it can scale to larger than memory data and now it can use multiple cores!
As an example, I could build views from 2 Pandas DataFrames with 2 columns and 1 million rows each, join them, and return the 1 million row dataset back to Pandas in 2 seconds vs. 40 seconds with SQLite/SQLAlchemy... Pretty sweet. DuckDB is going to be even faster now I bet!
[ Parent
| on HN ]
Re:DuckDB – An embeddable SQL database like SQLite, but supp...
by jmatthews on Sunday September 20, 2020 @08:43PM
(#24538705)
Have you compared this to dask from a performance standpoint? That is the larger than memory solution we're currently using for analytics.
Re:DuckDB – An embeddable SQL database like SQLite, but supp...
by chrismorgan on Sunday September 20, 2020 @01:02PM
(#24535232)
How about the other way? When will SQLite perform better than DuckDB?
Re:DuckDB – An embeddable SQL database like SQLite, but supp...
by FridgeSeal on Sunday September 20, 2020 @07:58PM
(#24538469)
If your workload is lots of singular INSERT or UPDATE’s.