Re:DuckDB – in-process SQL OLAP database management system
by corford on Friday February 10, 2023 @12:52PM
(#34743079)
This blog post offers a nice summary: https://motherduck.com/blog/six-reasons-duckdb-slaps/
[ Parent
| on HN ]
Re:DuckDB – in-process SQL OLAP database management system
by nerdponx on Friday February 10, 2023 @01:10PM
(#34743346)
The blog post doesn't really make a comparison between DuckDB and data frame libraries. It mentions that the DuckDB Python bindings can interoperate with Pandas, but it doesn't really explain why you would use DuckDB instead of Pandas, or Polars (which is both faster and more portable than Pandas).
Re:DuckDB – in-process SQL OLAP database management system
by simonw on Friday February 10, 2023 @05:45PM
(#34747037)
Don't Polars and Pandas both require your entire data to fit in memory?
Re:DuckDB – in-process SQL OLAP database management system
by nerdponx on Friday February 10, 2023 @06:37PM
(#34747577)
* * *
Re:DuckDB – in-process SQL OLAP database management system
by jcheng on Friday February 10, 2023 @07:46PM
(#34748220)
I think you mean that Pandas does require your entire data to fit in memory? https://pandas.pydata.org/docs/user_guide/scale.html
Re:DuckDB – in-process SQL OLAP database management system
by nerdponx on Friday February 10, 2023 @08:14PM
(#34748431)
Yes, thank you, that was a typo.
Re:DuckDB – in-process SQL OLAP database management system
by corford on Friday February 10, 2023 @09:32PM
(#34748986)
Apparently duckdb paired with arrow lets you work performatively on bigger than memory parquet data (I haven't tried this though)
Re:DuckDB – in-process SQL OLAP database management system
by riezebos on Saturday February 11, 2023 @03:31AM
(#34750618)
Arrow seems to be the common denominator for larger than ram. Polars is completely built on arrow, so the same advantages should apply.
https://github.com/pola-rs/polars#handles-larger-than-ram-da...