News for hackers. Stuff that matters.
Re:CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster tha... by mlthoughts2018 on Sunday October 11, 2020 @07:58PM (#24750172)
> “ Do you really believe that "parse CSV to a data frame" and "parse CSV to a parquet file" and "parse CSV and apply a query to it in-place" are completely separate, unrelated things?“

Yes, absolutely. Very high level design goals are completely different between those use cases and you would be forced to make mutually exclusive trade offs to solve any one of those problems - which will directly lead to concessions in terms of the other use cases. No one generic csv parser can be optimal to all three.

Re:CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster tha... by StefanKarpinski on Sunday October 11, 2020 @08:22PM (#24750315)
CSV.jl seems to be proof that you're wrong about that. More concretely, the only difference between parsing to a data frame, parsing to a parquet file, and executing a query over data is a) where you put the data in memory and b) optionally calling some code to operate on the data as you go. It's hard to see how those applications require radically different designs.
Re:CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster tha... by mlthoughts2018 on Monday October 12, 2020 @06:13PM (#24760050)
This is not accurate. CSV.jl optimizes for speed but does not optimize for data transfer to other systems like Spark (which, by necessity, causes worse performance in order to add features around type marshaling).

Design for general case speed is directly at odds with designing either for executing SQL over top of the file (in which case the trade offs are about indexing the data for record-wise or column-wise operations) or type marshaling for an extremely narrow use case like reading into Spark.

For example, parsing for speed vs parsing for Spark interop are very likely to handle datetime parsing or null parsing significantly differently.