A love letter to the CSV format

Re:A love letter to the CSV format by nomel on Wednesday March 26, 2025 @04:33PM (#43486966)

Oh, this is interesting. Are you tying different systems together? If so, do you use some preferred intermediate format? Do you have a giant library of * -> intermediate -> * converters that you sprinkle between everything? Or maybe the intermediate format is in memory?

What about Parquet and the like?

[ Parent | on HN ]

Re:A love letter to the CSV format by gopher_space on Wednesday March 26, 2025 @05:04PM (#43487301)

Not the person you were replying to, but from my experience CSV is a good place to define data types coming into a system and a safe way to dump data as long as I write everything down.

So I might do things like have every step in a pipeline begin development by reading from and writing to CSV. This helps with parallel dev work and debugging, and is easy to load into any intermediate format.

> do you use some preferred intermediate format?

This is usually dictated by speed vs money calculations, weird context issues, and familiarity. I think it's useful to look at both "why isn't this a file" and "why isn't this all in memory" perspectives.

[ Parent | on HN ]

Re:A love letter to the CSV format by theLiminator on Wednesday March 26, 2025 @05:29PM (#43487555)

For tabular/time-series greater than 100k rows I personally feel like parquet cannot be beat. It's self-describing, strongly-typed, relatively compact, supports a bunch of io/decode skipping, and is quite fast.

Orc also looks good, but isn't well supported. I think parquet is optimal for now for most analytical use-cases that don't require human readability.

[ Parent | on HN ]

Re:A love letter to the CSV format by Someone1234 on Wednesday March 26, 2025 @05:59PM (#43487899)

It is an interchange format, so it is inter-system by virtue of that. If I am a self-creator/consumer the format I use can be literally anything even binary memory dumps.

[ Parent | on HN ]