To understand how awful it is, you just have to ask two questions "How do I represent empty values?" "What do I do when my separator is contained in the data?"
There are answers to these questions, but there's not a standard backing up those answers. And that's what makes CSV is PITA to deal with. It's such a loose "standard" that anything goes.
xml and json are FAR better options even when you just want a table of data.
Those are rookie questions. What about when the data contains newline/crlf? What if the data contains the quote character and newline?
And why is the file mostly Windows-1252 encoded except some fields that are sometimes, at random, UTF-8 encoded?
The real kicker is when your fellow users are opening the CSV in a spreadsheet with a locale that prefers commas for fractional currency amount and then saving the file back.
https://en.wikipedia.org/wiki/CDATA is something far too few people pay attention to when considering the complications of object notation and why markup languages have some escape hatches.
,,
And yeah, dealing with separators is annoying, but pretty much every reader supports quoted values and escapes (not every writer cares up add them, though).
In practice I use csv to process large amount of tabular data (like logs, events, etc) where I care about greppability and performance more than about potential for 0.001% of corrupted data. YMMV of course, if getting it 100% right is important use something else (JSON is not without sins too, consider that JSON numbers are often parsed as floats).
Or is it ,,,,,,,
Or is it tabtabtabtab
Or perhaps it's "NULL,NULL,NULL"
My part of my work is data ingestion and I've seen all these (and more) as answers to the "empty values" question.
I'm not saying that other formats aren't without their problems, they certainly are. However, CSV doesn't just have those problems, it has multiple other problems on top of them.
It's a basic idea with really obvious edge cases addressed in multiple ways depending on who is producing these documents.
Empty string. For example, here are three in a row:
,,
> What do I do when my separator is contained in my values?Use quotes. For example, here is a single comma:
","I'm sorry about the title. As I said below; that was really meant in the spirit of fun.
On the subject of csvbase's content negotiation - yes that is an API. That was covered here some time ago when I wrote about it before: https://news.ycombinator.com/item?id=37526047
The "no API" bit I'm talking about in this article is basically the "trick" (or whatever word you want to use) of avoiding having any user-facing interface and just hooking into stuff that is already there. There is no "API surface" here for the user to learn beyond a url scheme. I think that's nice. And it's mainly what I'm talking about.
> Also, while CSV is a nice/convenient data format for a data analytics use case (like this), it’s certainly not a format I’d choose for an API where clients are likely to be more standard CRUD-ish apps. JSON is great for those, CSVs (with their trickier parsing, “everything is a string” data types, and enforced flatness) are a pain in the ass.
Without wanting to sound too much like a sales pitch: csvbase does offer JSON. Try https://csvbase.com/calpaterson/opcodes-6502.jsonl for JSON lines (or https://csvbase.com/calpaterson/opcodes-6502.json (no 'l') for a paged plain-JSON interface).
I personally think there is no ideal format for this at the moment. JSON is very very large and slow to parse. CSV has well known problems though has massive compatibility and often works well in practice. Parquet is probably closest to the ideal and excellent in many respects but is quite complicated to parse (moreso than CSV? perhaps) and anyway is effectively unstreamable - actually quite annoying for something like csvbase where you really _don't_ want to materialise the dataset while serving it.
> I did think it was interesting to learn about fsspec, didn’t know about that!
Yes it is cool isn't it. Millions of downloads, terabytes of bandwidth of PyPI and no one has heard of it.
There's no need to apologize! I and many others understood exactly what you meant by "no API". Some other people didn't understand the distinction you were drawing and chose to interpret their lack of understanding as you misleading them somehow, but that's on them, not you.
It was a great article that I thoroughly enjoyed! Thanks for sharing!
The python library has no python API, serving instead just as a plugin for fsspec. You then use pandas or anything else the same as you did before, the python library adds no (python) API. i guess technically "use the custom `csvbase://` scheme in the URIs you supply as input, instead of `http`" could be called an "API" if you really want to play gotcha.
I think the point is legit -- the python programmer has to reference nothing specific to the client library here other than the custom URI scheme, to then use remote data from the site with any one of several existing python data libraries, via their own python APIs.
Also, while CSV is a nice/convenient data format for a data analytics use case (like this), it’s certainly not a format I’d choose for an API where clients are likely to be more standard CRUD-ish apps. JSON is great for those, CSVs (with their trickier parsing, “everything is a string” data types, and enforced flatness) are a pain in the ass.
I did think it was interesting to learn about fsspec, didn’t know about that! And this style of client library does seem like a good/convenient one for this specific Python data analysis use case. Enjoyed that part of the article, but had to wade through a fair bit of clickbait style writing to get there.