News for hackers. Stuff that matters.
Re:Why isn’t there a decent file format for tabular data? by LeonB on Tuesday May 03, 2022 @08:02PM (#31255231)
One major limitation with quoted values that can this contain record delimiters (as opposed to escaping the delimiters) is that it stops systems from being able to load records in parallel.

Some systems ban embedded record delimiters, for this reason.

Btw, I’ve (previously) included at least one of your essays in “awesome csv” list at GitHub. https://github.com/secretGeek/AwesomeCSV#essays

There’s a few specs mentioned there too — is one of those the spec you worked on?

Re:Why isn’t there a decent file format for tabular data? by wahern on Tuesday May 03, 2022 @08:14PM (#31255313)
What's the difference between quoted and escaped delimiters? (Keeping in mind that escaping sequences can themselves be escaped, ad infinitum. You can't simply seek to an escape sequence and depend algorithmically on a small, fixed lookbehind.)
Re:Why isn’t there a decent file format for tabular data? by dataflow on Tuesday May 03, 2022 @08:18PM (#31255349)
I think the parent that if newlines were encoded as "\n" (with a backslash) then you could always split on (actual) newlines and process them in parallel without having to tokenize the quote first.
Re:Why isn’t there a decent file format for tabular data? by LeonB on Tuesday May 03, 2022 @11:44PM (#31256501)
Yep that’s exactly it.
Re:Why isn’t there a decent file format for tabular data? by dataflow on Tuesday May 03, 2022 @08:11PM (#31255289)
I think you still can load in parallel, but it just introduces potential for backtracking/correcting the data if you speculate incorrectly.