See, one of the reasons CSV managed to get so ubiquitous is precisely because all those things are unspecified. CSV is not a popular format; CSV is the name we give 960 visually similar but very different formats that as a collective are popular.
The thing you use CSV for is not it's technical merit. You use CSV for its ubiquity. If you nailed down all those things you talk about, you would have a much, much smaller user base and there would be no reason to use CSV in the first place.
(Hey, this reminds me of a similar situation governing s/CSV/C/g...)
Re:Dangers of CSV Injection
by johnwilkesbooth on Tuesday October 10, 2017 @03:21PM
(#15444588)
> No distinction between categorical data and strings. R thinks your strings are categories, and Pandas thinks your categories are strings.
I think this is more of an R-ism than a standardization issue. Strings are a pretty universal data type, where as categorical data (factors) are mostly specific to the domain of statistical modeling. IMO Python is doing the correct thing here. Personally I find factors to be more trouble than they are worth, and fortunately `data.table::fread` mimics Python in this regard.
.parquet fam, it's all about columnar data stores now
XKCD covered this: https://xkcd.com/927/
Some things I dislike about CSV:
* No distinction between categorical data and strings. R thinks your strings are categories, and Pandas thinks your categories are strings.
* I'm not a fan of the empty field. Pandas thinks it's a floating point NaN, while R doesn't. So is it a NaN? Is it an empty string? Does it mean Not Applicable? Does it mean Don't Know? Maybe it should be removed altogether.
* No agreement about escape characters.
* No agreement about separator characters.
* No agreement about line endings.
* No agreement about encoding. Is it ASCII, or UTF-8, or UTF-16, or Latin-whatever?
* None of the choices above are made explicit in the file itself. They all have the same extension "CSV".
These use up a bit of time whenever I get a CSV from a colleague, or even when I change operating system. Sometimes I end up clobbering the file itself.
Good things: * Human readable. * Simple.
I think the addition of some rules, and a standard interpretation of them, could go some way to improving the format.