Why Polars rewrote its Arrow string data type

Re:Why Polars rewrote its Arrow string data type by kristianp on Tuesday August 06, 2024 @11:34PM (#41178019)

Don't you lose the in-memory interop with other libraries by doing this? I'm thinking that duckdb will no longer be able to read polars data that has been loaded into memory, as it can currently do, due to duckdb supporting Arrow. Isn't the benefit of arrow that it's supported by many languages and libraries as a standard?

Will there be an option to use the "compatible" string format?

[ Parent | on HN ]

Re:Why Polars rewrote its Arrow string data type by emidln on Tuesday August 06, 2024 @11:37PM (#41178033)

Wouldnt you just recompile your duckdb with the hacked Arrow?

[ Parent | on HN ]

Re:Why Polars rewrote its Arrow string data type by suremarc on Wednesday August 07, 2024 @02:53PM (#41184187)

Arrow is a cross-language standard. Polars is written in Rust but DuckDB is written in C++. So it's not that simple unfortunately.

[ Parent | on HN ]

Re:Why Polars rewrote its Arrow string data type by emidln on Thursday August 08, 2024 @12:25PM (#41193271)

If you're using Arrow from duckdb, you're using C or C++ bindings. Hack the arrow lib, recompile (and adjust if necessary) whatever bindings/programs using your hacked arrow. If you're using multiple implementations of arrow that arent just bindings, you might have to hack it in more than one place.

You don't have to solve the entire world, you only need to hack what you actually use. If the issue with an optimized arrow impl is memory incompat in duckdb, recompile duckdb with your hacked arrow. This is where "overengineered" stuff like mono repos really shine. I've done exactly this type of thing at past jobs.

Don't let perfect be the enemy of good enough.

[ Parent | on HN ]

Re:Why Polars rewrote its Arrow string data type by mau on Wednesday August 07, 2024 @02:57AM (#41178870)

Arrow supports this string format: https://arrow.apache.org/docs/format/Columnar.html#variable-...

From the article:

> As luck would have it, the Arrow spec was also finally making progress with adding the long anticipated German Style string types to the specification. Which, spoiler alert, is the type we implemented.

[ Parent | on HN ]

Re:Why Polars rewrote its Arrow string data type by ritchie46 on Wednesday August 07, 2024 @06:40AM (#41179980)

DuckDB has their own string type (quite similar to this) that deviates from Arrow (large)-string type, so it had to do a copy anyway. Nothing has changed on that front.

[ Parent | on HN ]