News for hackers. Stuff that matters.
Re:DuckDB – An embeddable SQL database like SQLite, but supp... by legg0myegg0 on Sunday September 20, 2020 @12:00PM (#24534721)
Before the latest optimization, and only using 1 core, vs. SQLite we were seeing 133x performance on a basic group by or join, and about 4x for a pretty complex query. It was roughly even to Pandas in performance, but it can scale to larger than memory data and now it can use multiple cores! As an example, I could build views from 2 Pandas DataFrames with 2 columns and 1 million rows each, join them, and return the 1 million row dataset back to Pandas in 2 seconds vs. 40 seconds with SQLite/SQLAlchemy... Pretty sweet. DuckDB is going to be even faster now I bet!
Re:DuckDB – An embeddable SQL database like SQLite, but supp... by jmatthews on Sunday September 20, 2020 @08:43PM (#24538705)
Have you compared this to dask from a performance standpoint? That is the larger than memory solution we're currently using for analytics.
Re:DuckDB – An embeddable SQL database like SQLite, but supp... by chrismorgan on Sunday September 20, 2020 @01:02PM (#24535232)
How about the other way? When will SQLite perform better than DuckDB?
Re:DuckDB – An embeddable SQL database like SQLite, but supp... by FridgeSeal on Sunday September 20, 2020 @07:58PM (#24538469)
If your workload is lots of singular INSERT or UPDATE’s.