News for hackers. Stuff that matters.
Re:Show HN: ZSV (Zip Separated Values) columnar data format by orthoxerox on Saturday April 13, 2024 @06:35PM (#40026806)
I was comparing it with Parquet, which is much more complex, but has features that help you access the data in less than O(n), like row groups and pages.
Re:Show HN: ZSV (Zip Separated Values) columnar data format by cm2187 on Saturday April 13, 2024 @09:12PM (#40027810)
you mentioned NLJSON and CSV, which would require to read all columns from the disk.
Re:Show HN: ZSV (Zip Separated Values) columnar data format by orthoxerox on Sunday April 14, 2024 @03:44AM (#40029427)
Yes, but you would usually have to read at least two columns anyway. What are the datasets that are too large to be ingested completely, but too small for a proper columnar format?

If ZSV is meant to occupy the gap between CSV/NLJSON (smaller datasets) and Parquet/DuckDB (larger datasets), this niche is actually really small, if not nonexistent.

Re:Show HN: ZSV (Zip Separated Values) columnar data format by cm2187 on Sunday April 14, 2024 @04:13AM (#40029557)
yes it's unclear to me what is the advantage over parquet with compression. And there are enough file formats flying around already.