News for hackers. Stuff that matters.
Re:CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster tha... by mlthoughts2018 on Monday October 12, 2020 @06:13PM (#24760050)
This is not accurate. CSV.jl optimizes for speed but does not optimize for data transfer to other systems like Spark (which, by necessity, causes worse performance in order to add features around type marshaling).

Design for general case speed is directly at odds with designing either for executing SQL over top of the file (in which case the trade offs are about indexing the data for record-wise or column-wise operations) or type marshaling for an extremely narrow use case like reading into Spark.

For example, parsing for speed vs parsing for Spark interop are very likely to handle datetime parsing or null parsing significantly differently.

There are no comments on this comment yet.