News for hackers. Stuff that matters.

A comment on Working with CSV files on shell/terminal. Parent comment

Re:Working with CSV files on shell/terminal by elmolino89 on Wednesday June 28, 2023 @03:53AM (#36503601)
xsv is great for a quick sanity checks (i.e. number of columns, unique values counts in a given column) but for a more serious tasks/giant files I switch to either polars or duckdb converting CSV/TSV files to parquet or parquet data sets.

By giant I mean 25G gzipped files with >10^9 rows like these VCFs: https://ftp.ncbi.nlm.nih.gov/snp/latest_release/VCF/

[ Parent | on HN ]
Re:Working with CSV files on shell/terminal by rzmk on Thursday June 29, 2023 @03:44PM (#36525732)
Maybe try the to [1] and sqlp [2] commands from the qsv fork. From the README:

sqlp: Run blazing-fast Polars SQL queries against several CSVs - converting queries to fast LazyFrame expressions, processing larger than memory CSV files.

to: Convert CSV files to PostgreSQL, SQLite, XLSX, Parquet and Data Package.

[1] https://github.com/jqnatividad/qsv/blob/master/src/cmd/sqlp....

[2] https://github.com/jqnatividad/qsv/blob/master/src/cmd/to.rs...

[ Parent | on HN ]