Re:Doing a database join with CSV files
by faizshah on Tuesday December 31, 2019 @09:06PM
(#21926173)
When I've had to do similar, the easiest and fastest way has always been install spark -> Create an external spark sql table from the csvs -> use spark sql to join and aggregate/write. If you haven't used spark before it sounds complicated but its actually really quick and easy.
In general the operation on my laptop can get up to 200 MB/s, it's basically IO limited to the SSD of your machine.
In general the operation on my laptop can get up to 200 MB/s, it's basically IO limited to the SSD of your machine.