News for hackers. Stuff that matters.

A comment on DuckDB – in-process SQL OLAP database management system. Parent comment

Re:DuckDB – in-process SQL OLAP database management system by nerdponx on Friday February 10, 2023 @06:37PM (#34747577)
* * *
[ Parent | on HN ]
Re:DuckDB – in-process SQL OLAP database management system by jcheng on Friday February 10, 2023 @07:46PM (#34748220)
I think you mean that Pandas does require your entire data to fit in memory? https://pandas.pydata.org/docs/user_guide/scale.html
[ Parent | on HN ]
Re:DuckDB – in-process SQL OLAP database management system by nerdponx on Friday February 10, 2023 @08:14PM (#34748431)
Yes, thank you, that was a typo.
[ Parent | on HN ]
Re:DuckDB – in-process SQL OLAP database management system by corford on Friday February 10, 2023 @09:32PM (#34748986)
Apparently duckdb paired with arrow lets you work performatively on bigger than memory parquet data (I haven't tried this though)
[ Parent | on HN ]
Re:DuckDB – in-process SQL OLAP database management system by riezebos on Saturday February 11, 2023 @03:31AM (#34750618)
Arrow seems to be the common denominator for larger than ram. Polars is completely built on arrow, so the same advantages should apply.

https://github.com/pola-rs/polars#handles-larger-than-ram-da...

[ Parent | on HN ]