Re:Loading CSV File at the Speed Limit of the NVMe Storage
by NelsonMinar on Saturday October 10, 2020 @10:29AM
(#24739211)
Neat! I worked on a project where CSV parsing was a significant bottleneck (https://openaddresses.io/, it processes street addresses). Roughly 50% of the processing in the time is spent parsing CSV and decoding to Unicode. I dinked around a bit and tuned it to read at about 30 MB/second and figured that was good enough. So this is a 100x speedup in reading (assuming all the stars align) which would cut that 50% down to nearly zero. Just as soon as it works transparently with a single Python import statement I'll use it ;-)
[ Parent
| on HN ]
Re:Loading CSV File at the Speed Limit of the NVMe Storage
by whitten on Saturday October 10, 2020 @07:22PM
(#24742643)
Would a change of representation (away from CSV) significantly change parsing speed ?
Re:Loading CSV File at the Speed Limit of the NVMe Storage
by ingalls on Tuesday October 13, 2020 @12:56PM
(#24767402)
That's a good question! The new V4 platform https://batch.openaddresses.io/data outputs data as line delimited GeoJSONs, but haven't gotten time to test throughput in any common tools other than jq