News for hackers. Stuff that matters.
Re:Show HN: ZSV (Zip Separated Values) columnar data format by orthoxerox on Saturday April 13, 2024 @05:51PM (#40026484)
It is simple, but how do you access the price in row #1234567890? If your data doesn't have this many records and can fit into RAM, a basic NLJSON or CSV will work just as well.
Re:Show HN: ZSV (Zip Separated Values) columnar data format by cm2187 on Saturday April 13, 2024 @06:24PM (#40026709)
Like parquet this isn't really meant for RDBMS type of database, more like for analytics over large datasets. I work in an environment where we typically have tables with over 300 columns, 10s if not 100s millions of rows daily. When you want to do a simple sum/group by involving 2 or 3 columns, it is great to have a column store file format, where you only read the columns you need and those are compressed.

The price you pay is that it is inefficient for single record access, or for "select * " kind of queries.

Re:Show HN: ZSV (Zip Separated Values) columnar data format by orthoxerox on Saturday April 13, 2024 @06:35PM (#40026806)
I was comparing it with Parquet, which is much more complex, but has features that help you access the data in less than O(n), like row groups and pages.
Re:Show HN: ZSV (Zip Separated Values) columnar data format by cm2187 on Saturday April 13, 2024 @09:12PM (#40027810)
you mentioned NLJSON and CSV, which would require to read all columns from the disk.
Re:Show HN: ZSV (Zip Separated Values) columnar data format by orthoxerox on Sunday April 14, 2024 @03:44AM (#40029427)
Yes, but you would usually have to read at least two columns anyway. What are the datasets that are too large to be ingested completely, but too small for a proper columnar format?

If ZSV is meant to occupy the gap between CSV/NLJSON (smaller datasets) and Parquet/DuckDB (larger datasets), this niche is actually really small, if not nonexistent.

Re:Show HN: ZSV (Zip Separated Values) columnar data format by cm2187 on Sunday April 14, 2024 @04:13AM (#40029557)
yes it's unclear to me what is the advantage over parquet with compression. And there are enough file formats flying around already.
Re:Show HN: ZSV (Zip Separated Values) columnar data format by makmanalp on Saturday April 13, 2024 @09:22PM (#40027865)
Even with an OLAP use case, you're most often not scanning every row in the database if you even have a single where clause / conditional filter which is almost always. You need to have some level of locality and if your format doesn't support that, that'll be enough to kill performance.

Also parquet has lots of features that'll get you to the general vicinity of a single record tolerably fast without sacrificing much in terms of storage or computational complexity. It's a small price for a big win.

Re:Show HN: ZSV (Zip Separated Values) columnar data format by CapitalistCartr on Saturday April 13, 2024 @05:55PM (#40026506)
What is NLJSON?
Re:Show HN: ZSV (Zip Separated Values) columnar data format by chuckadams on Saturday April 13, 2024 @05:58PM (#40026534)
Also known as JSONL, or JSON Lines. Basically a file of JSON objects separated by newlines. Popular format for logs these days for obvious reasons.
Re:Show HN: ZSV (Zip Separated Values) columnar data format by eichin on Saturday April 13, 2024 @07:53PM (#40027338)
https://jsonlines.org/ was the first "this is trivial but let's write it down so maybe the name will stick" spec for it (from 2013ish)
Re:Show HN: ZSV (Zip Separated Values) columnar data format by mirekrusin on Sunday April 14, 2024 @03:40AM (#40029408)
Missed opportunity to just call it JSONS.
Re:Show HN: ZSV (Zip Separated Values) columnar data format by rzzzt on Saturday April 13, 2024 @06:41PM (#40026850)
NDJSON is the shorthand I've seen: https://github.com/ndjson/ndjson-spec
Re:Show HN: ZSV (Zip Separated Values) columnar data format by svieira on Saturday April 13, 2024 @05:57PM (#40026526)
New Line delimited JSON
Re:Show HN: ZSV (Zip Separated Values) columnar data format by hafthor on Saturday April 13, 2024 @09:23PM (#40027874)
There's two ways to limit the number of column-rows you have to read. One is by file partitioning, that is having many ZSV files rather than one giant one, ideally organized by partitioning key field(s). The other way is mentioned as an extension to the format itself which functions much like rowgroups do in Parquet. https://github.com/Hafthor/zsvutil?tab=readme-ov-file#row-gr...

Thanks for taking a look.

Re:Show HN: ZSV (Zip Separated Values) columnar data format by orthoxerox on Sunday April 14, 2024 @04:02AM (#40029500)
Oh, sorry, I must've missed the part about rowgroups and metadata. Yes, this should work to limit the scans to a reasonable amount.