News for hackers. Stuff that matters.
Re:Farewell Pandas, and thanks for all the fish by zhangyt26 on Friday August 30, 2024 @12:31AM (#41397736)
How does ibis manage ordering of the data frame for SQL based engine?

One big gap that I noticed is that pandas retain ordering between operations, bust most SQL based engine doesn’t give you that guarantee. Looking at duckdb, it looks they only do that for csv.

Re:Farewell Pandas, and thanks for all the fish by codyvoda on Friday August 30, 2024 @07:56AM (#41399817)
this is a huge difference between Ibis (and really, any dataframe library that works with larger-than-RAM or distributed engines) and pandas. PySpark, Polars, and Snowpark all to some extent also do not implicitly order your data

so in Ibis, you need to explicitly manage order with an `order_by` call at the end. there are a lot of technical reasons for why maintaining order is inefficient as you scale up