Re:Embedding user-defined indexes in Apache Parquet
by jasim on Tuesday July 15, 2025 @02:07AM
(#44568295)
I'm not sure if this is what you're looking for, but there is a proposal in DataFusion to allow user defined types. https://github.com/apache/datafusion/issues/12644
Re:Embedding user-defined indexes in Apache Parquet
by lmeyerov on Tuesday July 15, 2025 @11:51AM
(#44572422)
Thank you, looking forward to reading!
Re:Embedding user-defined indexes in Apache Parquet
by alamb on Tuesday July 15, 2025 @06:52AM
(#44569838)
We are actively working on supporting extension types. The mechanism is likely to be using the Arrow extension type mechanism (a logical annotation on top of existing Arrow types https://arrow.apache.org/docs/format/Columnar.html#format-me...)
I expect this to be used to support Variant https://github.com/apache/datafusion/issues/16116 and geometry types
(note I am an author)
Likewise, this means, afaict, it's likewise pretty straightforward to do novel indexing schemes within Iceberg as well just by reusing this.
The other aspect I've been curious about is the happy path pluggable types for custom columns. This shows one way, but I'm unclear if same thing.