Re:Working with CSV files on shell/terminal
by theLiminator on Wednesday June 28, 2023 @04:01PM
(#36511801)
I've found that for tabular data parquet can't be beat, the only time textual formats are superior are for human readabilty (which imo can be solved by tooling)
[ Parent
| on HN ]
Re:Working with CSV files on shell/terminal
by hermitcrab on Wednesday June 28, 2023 @04:03PM
(#36511822)
It is useful to be able to create, view or edit data in a standard text editor though.
Re:Working with CSV files on shell/terminal
by theLiminator on Wednesday June 28, 2023 @04:06PM
(#36511863)
Edits might be a little tricky, there's no concept of writing in place in parquet afaict. But there's absolutely no reason why reads/creation can't be supported in text editors (I'm sure some do).
Re:Working with CSV files on shell/terminal
by hermitcrab on Wednesday June 28, 2023 @04:15PM
(#36511955)
I often has to create small test data files a few rows long. Very easy to do for a CSV file.
Absolutely.
This is also my use case where I have GBs of data in S3 as Parquet and a small CSV file that I have to join them to.
And DuckDB reads CSVs and Parquet and SQLite and others. I can join all these heterogeneous data types in a single SQL statement and have the assurance that it’ll be done correctly.
I believe clickhouse-local can do the same.