News for hackers. Stuff that matters.
Re:Sidehelm: a pipeline to validate, test, and pull CSV data by victorNicollet on Thursday January 19, 2017 @06:10PM (#13440330)
It really is not that hard or time-consuming to write a CSV sanitization tool, one that actually manages to recover data from many kinds of bad (but consistent) formatting. I'd say my team spent around 6 man-weeks on it.

The real cost is that the fight is never over. Whenever a new customer comes in with a CSV so badly formatted that it is rejected by your system, further analysis reveals that there's actually some sanity behind the madness. That you could in fact add new rules to detect and correct the situation.

We've seen so many horrors. European dates ? Try a file with three different date formats over six date columns. Quotes that should be parsed in some columns but kept verbatim in others. Columns that contain raw binary data, without any escaping (but you can still infer the boundaries from context). UTF-16 with a byte skipped in the middle. Tab-separated files where the first line is space-separated instead. Cells that are 8MB of white space.

There is no end to the... creativity of CSV file authors.

Re:Sidehelm: a pipeline to validate, test, and pull CSV data by kafkaesq on Thursday January 19, 2017 @09:11PM (#13441242)
There is no end to the... creativity of CSV file authors.

Indeed, it never ceases to amaze. That's why it's so hard to imagine a "general" solution to this problem. You can think of something to handle to 90% most likely business use cases, maybe... but not the fat tail.

Re:Sidehelm: a pipeline to validate, test, and pull CSV data by iamwil on Friday January 20, 2017 @01:33PM (#13445844)
Sounds like it was painful. I actually laughed out loud at 8MB of white space. Mind if I learn more about your experiences? I don't know how to reach you, but please msg me at [email protected].
Re:Sidehelm: a pipeline to validate, test, and pull CSV data by jimktrains2 on Friday January 20, 2017 @10:23AM (#13444406)
> There is no end to the... creativity of CSV file authors.

Eh, commas and new lines are common, let's use pipes. and count fields.

:'(