News for hackers. Stuff that matters.

A comment on FireDucks: Pandas but 100x Faster. Parent comment

Re:FireDucks: Pandas but 100x Faster by rapatel0 on Wednesday November 20, 2024 @11:30AM (#42195551)
Look into [Ibis](https://ibis-project.org/). It's a dataframe library built on duckdb. It supports lazy execution, greater than memory datastructures, remote s3 data and is insanely fast. Also works with basically any backend (postgres, mysql, parquet/csv files, etc) though there are some implementation gaps in places.

I previously had a pandas+sklearn transformation stack that would take up to 8 hours. Converted it to ibis and it executes in about 4 minutes now and doesn't fill up RAM.

It's not a perfect apples to apples pandas replacement but really a nice layer on top of sql. after learning it, I'm almost as fast as I was on pandas with expressions.

[ Parent | on HN ]
Re:FireDucks: Pandas but 100x Faster by techwizrd on Wednesday November 20, 2024 @12:27PM (#42196216)
I made the switch to Ibis a few months ago and have been really enjoying it. It works with all the plotting libraries including seaborn and plotnine. And it makes switching from testing on a CSV to running on a SQL/Spark a one-line change. It's just really handy for analysis (similar to the tidyverse).
[ Parent | on HN ]