News for hackers. Stuff that matters.
Re:ParaText: CSV parsing at 2.5 GB per second by amelius on Wednesday June 08, 2016 @06:14AM (#11861187)
The main advantage of CSV is that it is human-readable. But if you have this much data, then why not use some binary format that you can just read in as a blob, and don't even need to parse?

Also, the approach mentioned here is only useful if reading+parsing the data is the bottleneck (or close to it). For example, if reading+parsing takes only 10% of the total processing time, then optimizing this stage will only give at most a 11% increase in performance.

Re:ParaText: CSV parsing at 2.5 GB per second by bladecatcher on Wednesday June 08, 2016 @06:59AM (#11861356)
Csv is an ubiquitous data format. Especially when you're receiving data from external sources which you don't have control over, which often happens to be the case in banking and finance (streaming csv dumps are quite common)
Re:ParaText: CSV parsing at 2.5 GB per second by Chyzwar on Wednesday June 08, 2016 @05:44PM (#11865937)
CSV is stupid format. XML, JSON, YAML are better to read and easier to parse/validate. After compression these formats takes similar amount of space or even less when cross-referencing.

CSV, tab-delimited files and in general flat files are retarded idea to be used in banking. I know because I work for bank, these are source of all misery.

Re:ParaText: CSV parsing at 2.5 GB per second by ovi256 on Wednesday June 08, 2016 @07:11AM (#11861413)
Adding to what bladecatcher said, even if you want to work internally with a binary format, the fact that you receive large csv datadumps from the outside means you'll have to transform them into your internal format of choice. The first part of that transformation would greatly benefit from a fast csv parser such as ParaText.

Another datapoint: data changes (price updates, number ranges) between European telecom operators are all csv files over ftp. You can recognize the security aware by their use of sftp!

Re:ParaText: CSV parsing at 2.5 GB per second by roel_v on Wednesday June 08, 2016 @09:28AM (#11862112)
"For example, if reading+parsing takes only 10% of the total processing time, then optimizing this stage will only give at most a 11% increase in performance."

10% performance increase is nothing to scoff at in a mature code base.