News for hackers. Stuff that matters.

A comment on ParaText: CSV parsing at 2.5 GB per second.

Re:ParaText: CSV parsing at 2.5 GB per second by justinsaccount on Tuesday June 07, 2016 @10:22PM (#11859569)
This is impressive, but...

"A fast reader exploits the capabilities of the storage system"...

the graphs show that their storage system is doing 4.00 GB/sec

I wonder what processor this is running on and what their storage system this is.. multiple PCIe SSD?

I tried running a quick test but only succeeded in OOMing my 8G laptop.

Even just doing

  import paratext
  it = paratext.load_csv_as_iterator("/dev/shm/tmp/c.log", expand=True, forget=True)
  x = it.next()
Starts eating up all my ram after about a minute of spinning the cpu... so I think they have a slightly different definition of an iterator as everyone else.

Compared to

  cut -d , -f 5 < c.log > /dev/null
which runs in a few seconds, or a slightly more domain specific and optimized version of 'cut'[1] that runs even faster (300-500MB/s on a single core depending on which fields you want)

  $ du -hs c.log ;wc -l c.log 
  2.1G	c.log
  16197412 c.log
I also wonder if that is 2.5 GB/s per core.

https://github.com/BurntSushi/rust-csv does 241 MB/s in raw mode, so I find it a little hard to believe that this is 10x faster... unless that is while maxing out multiple cores.

[1] https://github.com/bro/bro-aux/blob/master/bro-cut/bro-cut.c

[ Parent | on HN ]
Re:ParaText: CSV parsing at 2.5 GB per second by burntsushi on Wednesday June 08, 2016 @07:53AM (#11861554)
From looking at their source briefly, I can make two observations:

Firstly, their core parsing routine is a state machine not unlike the one found in rust-csv: https://github.com/wiseio/paratext/blob/master/src/csv/rowba...

Secondly, they do indeed appear to be achieving high parsing speed using coarsely grained parallelism. You can see the code that chunks up the CSV data here: https://github.com/wiseio/paratext/blob/master/src/generic/c...

It's pretty clever!

[ Parent | on HN ]
Re:ParaText: CSV parsing at 2.5 GB per second by Ygg2 on Wednesday June 08, 2016 @11:07AM (#11862711)
Can this be used to speed up rust-csv?
[ Parent | on HN ]
Re:ParaText: CSV parsing at 2.5 GB per second by burntsushi on Wednesday June 08, 2016 @04:28PM (#11865390)
It feels like something that should be built on top of rust-csv.
[ Parent | on HN ]
Re:ParaText: CSV parsing at 2.5 GB per second by Ygg2 on Thursday June 09, 2016 @07:45AM (#11868736)
I thought this was base ability of the parser to parse CSV in chunks.
[ Parent | on HN ]
Re:ParaText: CSV parsing at 2.5 GB per second by ygra on Wednesday June 08, 2016 @12:25AM (#11860012)
Does cut work with quoted fields correctly? My understanding was that it was just a dumb line tokenizer and CSV is a little more complex than that, e.g., CSV rows can span more than one line when fields contain line breaks.
[ Parent | on HN ]
Re:ParaText: CSV parsing at 2.5 GB per second by gkst on Wednesday June 08, 2016 @07:02AM (#11861370)
No, it doesn't and therefore is no viable option for many real-world datasets.
[ Parent | on HN ]
Re:ParaText: CSV parsing at 2.5 GB per second by rspeer on Wednesday June 08, 2016 @12:28AM (#11860023)
Which means an effective thing to do when you control the data pipeline is to ban tabs and line-breaks from your values, and then use `cut` on tab-separated files.
[ Parent | on HN ]
Re:ParaText: CSV parsing at 2.5 GB per second by ygra on Wednesday June 08, 2016 @02:24AM (#11860382)
Sure, if you can redefine your data format, you can take shortcuts. But otherwise comparing cut to a CSV parser is just a different way of saying »if it doesn't have to be correct, I can make it as fast as I want«.
[ Parent | on HN ]
Re:ParaText: CSV parsing at 2.5 GB per second by Twirrim on Wednesday June 08, 2016 @01:04AM (#11860144)
Or just not use CSV as a data transfer format :D
[ Parent | on HN ]
Re:ParaText: CSV parsing at 2.5 GB per second by rspeer on Wednesday June 08, 2016 @11:52AM (#11863060)
I think you're missing the point. Unix tools like `cut` are really effective ways to deal with plain text data, who cares if it's CSV.
[ Parent | on HN ]
Re:ParaText: CSV parsing at 2.5 GB per second by omgtehlion on Wednesday June 08, 2016 @03:55AM (#11860707)
> 4.00 GB/sec

> I wonder what processor this is running on and what their storage system this is.. multiple PCIe SSD?

Consumer grade NVMe disks currently achiveve 2GB/s. It is easy to use a couple of those, or just one professional SSD.

[ Parent | on HN ]
Re:ParaText: CSV parsing at 2.5 GB per second by IanCal on Wednesday June 08, 2016 @06:08AM (#11861171)
> Starts eating up all my ram after about a minute of spinning the cpu... so I think they have a slightly different definition of an iterator as everyone else.

I don't think so, the difference is you're expecting it to read one row at a time and it's actually reading one column at a time. load_csv_as_iterator is iterating over the columns.

[ Parent | on HN ]
Re:ParaText: CSV parsing at 2.5 GB per second by JoshTriplett on Wednesday June 08, 2016 @12:23AM (#11860005)
> I wonder what processor this is running on and what their storage system this is.. multiple PCIe SSD?

One high-end PCIe SSD can manage that; two easily could. A high-end NAS might, too.

[ Parent | on HN ]