Re:Show HN: ZSV (Zip Separated Values) columnar data format
by orthoxerox on Saturday April 13, 2024 @06:35PM
(#40026806)
I was comparing it with Parquet, which is much more complex, but has features that help you access the data in less than O(n), like row groups and pages.
[ Parent
| on HN ]
Re:Show HN: ZSV (Zip Separated Values) columnar data format
by cm2187 on Saturday April 13, 2024 @09:12PM
(#40027810)
you mentioned NLJSON and CSV, which would require to read all columns from the disk.
Re:Show HN: ZSV (Zip Separated Values) columnar data format
by orthoxerox on Sunday April 14, 2024 @03:44AM
(#40029427)
Yes, but you would usually have to read at least two columns anyway. What are the datasets that are too large to be ingested completely, but too small for a proper columnar format?
If ZSV is meant to occupy the gap between CSV/NLJSON (smaller datasets) and Parquet/DuckDB (larger datasets), this niche is actually really small, if not nonexistent.