News for hackers. Stuff that matters.
Re:Doing a database join with CSV files by faizshah on Tuesday December 31, 2019 @09:06PM (#21926173)
When I've had to do similar, the easiest and fastest way has always been install spark -> Create an external spark sql table from the csvs -> use spark sql to join and aggregate/write. If you haven't used spark before it sounds complicated but its actually really quick and easy.

In general the operation on my laptop can get up to 200 MB/s, it's basically IO limited to the SSD of your machine.

Re:Doing a database join with CSV files by lolive on Wednesday January 01, 2020 @06:07PM (#21931407)
Would be nice if you could write a small article about that.
Re:Doing a database join with CSV files by ecnahc515 on Wednesday January 01, 2020 @07:42AM (#21927813)
Similarly, I've used Prestoql for this. It's similar to Spark and Hive.