News for hackers. Stuff that matters.
Re:DuckDB – in-process SQL OLAP database management system by camgunz on Friday February 10, 2023 @07:58PM (#34748306)
Databases are just much, much faster than Pandas, and that's before you start factoring the extraction and loading of data. I treat Pandas as a last resort when I can't do something in SQL, generally this is something like integrating with external services or running recordlinkage.
Re:DuckDB – in-process SQL OLAP database management system by RobinL on Saturday February 11, 2023 @02:26AM (#34750340)
If you're curious, I've written a FOSS record linkage library that executes everything as SQL. It supports multiple SQL backends including DuckDB and Spark for scale, and runs faster than most competitors because it's able to leverage the speed of these backends: https://github.com/moj-analytical-services/splink
Re:DuckDB – in-process SQL OLAP database management system by camgunz on Saturday February 11, 2023 @05:02AM (#34750976)
Oh hot tip! Thank you! Love the blog btw