News for hackers. Stuff that matters.
Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by elmolino89 on Tuesday March 08, 2022 @11:15AM (#30601585)
I had to ingest and explore enough dirty CSV/TSV data to use CLI tools only to get a first glimpse what's there. Whenever facing anything non trivial I go for CSV to parquet, and preferably write intermediate parquet data sets. Then DuckDB / SQL queries for slicing and dicing. I am yet to encounter some readable to a newcomers awk/uniq/sed combos for intersecting bunch of CSVs with 30+ columns.

On the other hand parquet becomes a turtle if one tries to squeeze i.e. 12k numerical columns into it.

Re:Command-line Tools can be 235x Faster than your Hadoop Cl... by p33p on Tuesday March 08, 2022 @11:29AM (#30601798)
> On the other hand parquet becomes a turtle if one tries to squeeze i.e. 12k numerical columns into it.

I thought parquet was columnar stored? Is this a fault of parquet or just the shear number of columns trying to get accessed?

I agree with your general premise though. I'd rather take a dirty dataset, throw it into S3, spin up a Redshift cluster, do what I need, spin down the cluster. You can work with billions of records fairly easily with plain old SQL and c-store databases.