News for hackers. Stuff that matters.

A comment on CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R. Parent comment

Re:CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster tha... by boxed on Monday October 12, 2020 @12:58AM (#24751517)
Pandas is pretty slow and since it loads into memory it can be totally infeasible for even relatively small data sets. The csv module it what one should compare it to imo.
[ Parent | on HN ]
Re:CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster tha... by RobinL on Monday October 12, 2020 @03:47AM (#24752298)
Apache Arrow reads csvs into memory in Arrow format, not pandas format. They are independent libraries that do different things. In Arrow, it is possible to read a csv in batches, obviating memory problems. See 'incremental reading' - http://arrow.apache.org/docs/python/csv.html
[ Parent | on HN ]