News for hackers. Stuff that matters.
Re:Farewell Pandas, and thanks for all the fish by __mharrison__ on Thursday August 29, 2024 @12:20PM (#41392444)
Jump straight to cluster and skip Dask?

Not sure what "big" means here, but a combination of .pipe, pyarrow, and polars can speed up many slow Pandas operations.

Polars streaming is surprisingly good for larger than RAM. I get that clusters are cool, but I prefer to keep it on a single machine if possible.

Also, libraries like cudf can greatly speed up Pandas code on a single machine, while Snowpark can scale Pandas code to Snowflake scale.

Re:Farewell Pandas, and thanks for all the fish by cpcloud on Thursday August 29, 2024 @12:29PM (#41392554)
In my experience, Polars streaming runs out of memory at much smaller scales than both DuckDB and DataFusion and tends to use much more memory for the same workload when it doesn't outright segfault.

Polars is faster than those two once you get to less than a few GB, but beyond that you're better off with DuckDB or DataFusion.

I would love for this to improve in Polars, and I'm sure it will!

Re:Farewell Pandas, and thanks for all the fish by ritchie46 on Thursday August 29, 2024 @12:58PM (#41392914)
Do you mean segfault or OOM? I am not aware of Polars segfaulting on high memory pressure.

If it does segfault, would you mind opening an issue?

Some context; Polars is building a new streaming engine that will eventually be ready to run the whole Polars API (Also the hard stuff) in a streaming fashion. We expect the initial release end of this year/early next year.

Our in-memory engine isn't designed for out-of-core processing and thus if you benchmark it on restricted RAM, it will perform poorly as data is swapped or you go OOM. If you have a machine with enough RAM, Polars is very competitive in performance. And in our experience it is tough to beat in time-series/window functions.

Re:Farewell Pandas, and thanks for all the fish by cpcloud on Thursday August 29, 2024 @01:10PM (#41393017)
Segmentation violations are often the result of different underlying problems, one of which can be running out of memory.

We (the Ibis team) have opened related issues and the usual response is to not use streaming until it's ready, or to fix the problem if it can be fixed.

Not sure what else there is to do, seems like things are working as expected/intended for the moment!

We'll definitely be the first to try out any improvements to the streaming engine.

Re:Farewell Pandas, and thanks for all the fish by ritchie46 on Thursday August 29, 2024 @01:34PM (#41393278)
They have different implications for us. An abort due to an OOM isn't a bug in our program, as SEGFAULT is a serious bug we want to fix.
Re:Farewell Pandas, and thanks for all the fish by ledauphin on Thursday August 29, 2024 @11:20PM (#41397470)
when you say "our in-memory engine", are you talking about dataframe or the lazyframe?
Re:Farewell Pandas, and thanks for all the fish by ritchie46 on Friday August 30, 2024 @01:42AM (#41397978)
A DataFrame is our in memory table. A LazyFrame is a compute plan that can have DataFrames as source.

The engine is what executes our plans and materializes a result. This is plural as we are building a new one.

Re:Farewell Pandas, and thanks for all the fish by __mharrison__ on Thursday August 29, 2024 @12:32PM (#41392598)
My understanding is that the Polars team is working on a new streaming engine. It looks like you will get your wish.