News for hackers. Stuff that matters.

A comment on Show HN: Vortex – a high-performance columnar file format in Rust. Parent comment

Re:Show HN: Vortex – a high-performance columnar file format... by sa46 on Monday October 14, 2024 @07:21PM (#41843259)
Parquet also encodes the physical layout using footers [1], as does ORC [2]. Perhaps the author meant support for semi-structured data, like the spans you mention.

[1]: https://parquet.apache.org/docs/file-format/

[2]: https://orc.apache.org/specification/ORCv2/#file-tail

[ Parent | on HN ]
Re:Show HN: Vortex – a high-performance columnar file format... by danking00 on Monday October 14, 2024 @10:37PM (#41844396)
Yeah we should be more clear in our description about how our footers differ from Parquet. Parquet is a bit more prescriptive; for example, it requires row groups which are not required by Vortex. If you have a column with huge values and another column of 8 bit ints, they can be paged separately, if you like.
[ Parent | on HN ]