XSV: A fast CSV command-line toolkit written in Rust

Re:XSV: A fast CSV command-line toolkit written in Rust by hugh-avherald on Sunday September 09, 2018 @11:39AM (#17946377)

Have you seen the R package data.table (https://github.com/rdatatable/data.table)?

Not sure whether data.table is in the same domain as xsv, and certainly a lot of it is written in C. But for comparison's sake:

  fread("cities.csv")  1.30 s

And then the rest of the computations will be faster of course:

  count -- 0.005 ms
  freq  -- 0.2   s
  sort  -- 0.1   s

It's so useful that I often just use csvs between 10 and 100GB as a database as the difference in performance between fread and a 'proper' database aren't enough to justify the latter.

[ Parent | on HN ]

Re:XSV: A fast CSV command-line toolkit written in Rust by burntsushi on Sunday September 09, 2018 @11:47AM (#17946411)

Yes. I've used it lightly in the past and have never been impressed by its speed or memory usage. But there could have been user errors. I am not an R expert. :)

In any case, I think R implements all or most of its data table transformations in C, so I don't think it applies well here.

[ Parent | on HN ]

Re:XSV: A fast CSV command-line toolkit written in Rust by hugh-avherald on Sunday September 09, 2018 @11:55AM (#17946445)

I'm curious as to what has impressed you with regard to speed or memory usage (at least over this scale). I realize there's a lot I don't know.

[ Parent | on HN ]