Re:Show HN: ZSV (Zip Separated Values) columnar data format
by thehappypm on Saturday April 13, 2024 @07:38PM
(#40027226)
Seeking within a column
Re:Show HN: ZSV (Zip Separated Values) columnar data format
by hafthor on Sunday April 14, 2024 @11:52AM
(#40032011)
There's two ways to limit the number of column-rows you have to read. One is by file partitioning, that is having many ZSV files rather than one giant one, ideally organized by partitioning key field(s). The other way is mentioned as an extension to the format itself which functions much like rowgroups do in Parquet. https://github.com/Hafthor/zsvutil?tab=readme-ov-file#row-gr...
Thanks for taking a look.
> ZIP files are a collection of individually compressed files, with a directory as a footer to the file, which makes it easy to seek to a specific file without reading the whole file... The nature of .zip files makes it possible to seek and read just the columns required without having to read/decode the other columns.