News for hackers. Stuff that matters.
Re:One-liner for running queries against CSV files with SQLite by avogar on Tuesday June 21, 2022 @11:31AM (#31825093)
Using ClickHouse you can also process local files in one line using clickhouse-local command tool. And it will look a lot easier:

clickhouse local -q "SELECT passenger_count, COUNT(*), AVG(total_amount) FROM file(taxi.csv, 'CSVWithNames') GROUP BY passenger_count"

And ClickHouse supports a lot of different file formats both for import and export (you can see all of them here https://clickhouse.com/docs/en/interfaces/formats/).

There is an example of using clickhouse-local with taxi dataset mentioned in the post: https://colab.research.google.com/drive/1tiOUCjTnwUIFRxovpRX...

Re:One-liner for running queries against CSV files with SQLite by eatonphil on Tuesday June 21, 2022 @02:03PM (#31827084)
Clickhouse-local is incredible. It does the best of any similar tool I've benchmarked. But the reason I took it out of the linked benchmarks in OP's post is because it's 2+GB. That's a massive binary. It's the whole server. I'm not sure you want to be distributing this all over the place in general. It's just not in the same category IMO. Disclaimer: I build another tool that does similar things.
Re:One-liner for running queries against CSV files with SQLite by zX41ZdbW on Tuesday June 21, 2022 @02:36PM (#31827421)
It is shipped with debug info and symbol tables just in case.

Without debug info, it will be 350 MB, and compressed can fit in 50 MB: https://github.com/ClickHouse/ClickHouse/issues/29378

It is definitely a worth improvement.

Re:One-liner for running queries against CSV files with SQLite by eatonphil on Tuesday June 21, 2022 @04:02PM (#31828471)
Aha, thanks for the clarification and link. I'll be following that issue.
Re:One-liner for running queries against CSV files with SQLite by qoega on Tuesday June 21, 2022 @02:36PM (#31827415)
It is that big only with debug symbols. If you take packaged version of ClickHouse it will be much smaller. Or just strip large binary manually if you already have it.