News for hackers. Stuff that matters.
Re:Working with CSV files on shell/terminal by hermitcrab on Wednesday June 28, 2023 @03:02AM (#36503271)
I think you can use a heuristic approach. E.g. pick a comma in the middle of the CSV, assume it is a delimiter (not inside data) and parse from there in a separate thread. If you find it doesn't parse correctly (perhaps because it is inside data) you might have to throw the results of that thread away. Kind of ugly but would probably allow faster parsing in 99% of cases.

CSVs are a bit of a mess. Non-trivial to parse properly, due to escaping, and the standard (such as it is) is poorly followed. E.g. sometimes people use semi-colon or pipe instead of comma.

See also: 'Why isn’t there a decent file format for tabular data?' https://news.ycombinator.com/item?id=31220841

Re:Working with CSV files on shell/terminal by theLiminator on Wednesday June 28, 2023 @04:01PM (#36511801)
I've found that for tabular data parquet can't be beat, the only time textual formats are superior are for human readabilty (which imo can be solved by tooling)
Re:Working with CSV files on shell/terminal by hermitcrab on Wednesday June 28, 2023 @04:03PM (#36511822)
It is useful to be able to create, view or edit data in a standard text editor though.
Re:Working with CSV files on shell/terminal by theLiminator on Wednesday June 28, 2023 @04:06PM (#36511863)
Edits might be a little tricky, there's no concept of writing in place in parquet afaict. But there's absolutely no reason why reads/creation can't be supported in text editors (I'm sure some do).
Re:Working with CSV files on shell/terminal by hermitcrab on Wednesday June 28, 2023 @04:15PM (#36511955)
I often has to create small test data files a few rows long. Very easy to do for a CSV file.
Re:Working with CSV files on shell/terminal by wenc on Wednesday June 28, 2023 @05:10PM (#36512659)
Absolutely.

This is also my use case where I have GBs of data in S3 as Parquet and a small CSV file that I have to join them to.

And DuckDB reads CSVs and Parquet and SQLite and others. I can join all these heterogeneous data types in a single SQL statement and have the assurance that it’ll be done correctly.

I believe clickhouse-local can do the same.