News for hackers. Stuff that matters.
Re:Show HN: ZSV (Zip Separated Values) columnar data format by hafthor on Saturday April 13, 2024 @09:23PM (#40027874)
There's two ways to limit the number of column-rows you have to read. One is by file partitioning, that is having many ZSV files rather than one giant one, ideally organized by partitioning key field(s). The other way is mentioned as an extension to the format itself which functions much like rowgroups do in Parquet. https://github.com/Hafthor/zsvutil?tab=readme-ov-file#row-gr...

Thanks for taking a look.

Re:Show HN: ZSV (Zip Separated Values) columnar data format by orthoxerox on Sunday April 14, 2024 @04:02AM (#40029500)
Oh, sorry, I must've missed the part about rowgroups and metadata. Yes, this should work to limit the scans to a reasonable amount.