Re:New technique to easily partition CSV files for parallel ...
by Someone on Thursday February 26, 2026 @03:58AM
(#47163638)
> This is clearly not the sort of thing you should expose to anyone, it is an optimization technique.
If you’re optimizing a workflow that uses CSV files and the data in those files is under your control, speeding up parallel reading of CSV files wouldn’t be on my list; replacing CSV by, say, parquet would. An easy to implement alternative would be to simply write multiple CSV files.
If you’re optimizing a workflow that uses CSV files and the data in those files is under your control, speeding up parallel reading of CSV files wouldn’t be on my list; replacing CSV by, say, parquet would. An easy to implement alternative would be to simply write multiple CSV files.