Sure technically you can do things like having new lines on a line which cause all sorts of grief, but if you control the end to end workflow just avoid it.
I have a tool which monitors performance of loading various websites via various proxies at various points ok our network with pupetteer. That outputs a csv which I can then extract data from and make handy graphs to show manglement.
Yeah, I guess that kinda works if you're the one generating them. But you must not have too many strings -- like a name field containing "Smith, John" or a book title like "The Lion, the Witch and the Wardrobe"? When I deal with CSV's those kind of things are quote common. I actually prefer TSV because you can (more easily) process them with standard AWK.
Related example is the recently discussed problems with filenames, and the many pitfalls that scripts can stumble upon when handling them.
The thing is that from the point of view of each individual tool, there isn't even an error! You ask awk to split on commas and give you the second field, it splits on commas and gives you the second field. It opened the file no problem, it didn't run out of memory writing its output to stdout... There were no errors to speak of from what it could see. The "error" is an error in the overall logic of the code.
The problem is that edge cases are not really edge cases… in certain locales. ;) Someone could be using commas as the decimal separator since that’s how they do it ’round these parts.[1][2] So it’s not like they’re storing “weird” stuff in tabular format, like HTML blog articles; they’re just storing numbers.
Of course, Excel and other software has tried to be way too clever about it with different separators depending on locale, which makes the whole pipeline complex and context-dependent.
[1] A surprising amount of countries have this convention, including my own… doesn’t make sense to me since it becomes harder to list numbers.
[2] And the number one selling point of CSV is that it is “human editable”, so of course someone is going to do that at some point in the pipeline.
<https://github.com/dbro/csvquote>
Using it with the first example command from this article would be
csvquote file.csv | awk -F, '{print $1}' | csvquote -u
By using the "-u" flag in the last step of the pipeline, all of the problematic quoted delimiters get restored.https://stackoverflow.com/questions/43477652/ignoring-comma-...
The frawk tool (written in Rust) also supports this.
Interestingly, Brian Kernighan is currently updating the book "The AWK Programming Language" for a second edition (I'm one of the technical reviewers), and Gawk and awk are adding a "--csv" option for this purpose. So real CSV mode is coming to an AWK near you soon!