Re:Farewell Pandas, and thanks for all the fish
by codyvoda on Friday August 30, 2024 @07:56AM
(#41399817)
this is a huge difference between Ibis (and really, any dataframe library that works with larger-than-RAM or distributed engines) and pandas. PySpark, Polars, and Snowpark all to some extent also do not implicitly order your data
so in Ibis, you need to explicitly manage order with an `order_by` call at the end. there are a lot of technical reasons for why maintaining order is inefficient as you scale up
One big gap that I noticed is that pandas retain ordering between operations, bust most SQL based engine doesn’t give you that guarantee. Looking at duckdb, it looks they only do that for csv.