Re:DuckDB – in-process SQL OLAP database management system
by camgunz on Friday February 10, 2023 @07:58PM
(#34748306)
Databases are just much, much faster than Pandas, and that's before you start factoring the extraction and loading of data. I treat Pandas as a last resort when I can't do something in SQL, generally this is something like integrating with external services or running recordlinkage.
[ Parent
| on HN ]
Re:DuckDB – in-process SQL OLAP database management system
by RobinL on Saturday February 11, 2023 @02:26AM
(#34750340)
If you're curious, I've written a FOSS record linkage library that executes everything as SQL. It supports multiple SQL backends including DuckDB and Spark for scale, and runs faster than most competitors because it's able to leverage the speed of these backends: https://github.com/moj-analytical-services/splink