ASCII has special delimiters 0x1E Record Separator and 0x1F Unit Separator to avoid conflicting with values, but they have never gained widespread adoption.

> ASCII has special delimiters 0x1E Record Separator and 0x1F Unit Separator

And since ASCII has them, Unicode has them too.

While these have clear advantages over comma and CR for parsing (no more escaping!), they aren't at all convenient for manual editing.

If they were popular they'd be supported in your editor though.

Maybe. It is a bit of a chicken-and-egg problem though. They aren't likely to become popular until editors support them.

Also, the fact that they have been around for years and never taken off implies that they probably never will.

More generally than editor support, how is the average person expected to type them? I could easily add them to my keyboard, but if "editor support" turned out to mean "Edit > Insert > Record Separator Character" and I was now bound only to editors that support this, absolutely nobody is going to bother. They specifically need to be trivially typable in exactly the way commas and newlines are.

Realistically I don't think anyone (at least on HN) would presume that editor support would be so weirdly obtuse, but it's not exactly well-defined here, either.

I'm mostly just thinking out loud, here, not really critiquing anyone.

Commas and Newlines are only trivial if your data contains no commas or newlines.

Who's writing CSV files by hand though?

I'd expect some kind of CSV mode in my text editor where Tab and Newlines are turned into keys to insert those characters

> how is the average person expected to type them?

[modifier]-[C,D,E,F]

Actually, there is already a standard way for someone to type these with the control key. To be specific, they are:

  FS: ^\ GS: ^] RS: ^^ US: ^_

Since these delimiters are not much used, on many terminals ^\ is mapped to send SIGQUIT. Use 'stty quit undef' and you can then type this at the keyboard. This works on Linux and OS X, and likely most other systems. Note that with most keyboards you are also using shift for RS and US. For emacs, prefix with C-q so that it inserts the literal.

Other posters have noted that they have ASCII separators recently, I have as recently as 2018 internally at a FAANG. So they are not completely unknown in the wild.

which modifier would that be, Alt/Ctrl or Cmd? Also there is a chance that this combo will not work on some platforms, some may already be in use as a hotkey.

I worked on a project that used these in a proprietary file format. It was actually quite nice. There was no escaping required because they simply stripped those characters out of the actual content.

> but they have never gained widespread adoption.

Yes, because that just kicks the can down the road: how do you represent the one-byte string "\x1E"?

At least very few humans type that into their spreadsheet cell.

The other advantage not being discussed here is that once you have a csv parser, it can be trivially generalized to use any character as a delimiter, and you can usually find a character that isn't found in the data you're working with unless you're working with binary data. I work on a legacy system where passing delimited files around is usually the integration method of the day, and if you can't delimit with commas (common) then you just delimit with tabs. If you can't delimit with tabs or commas, then delimit with pipes, which are rarely if ever used outside of code. Or use tildes, or grave accents, or...

Doing this avoids quoting, which avoids an entire class of problems. And yes, if you're passing multi-megabyte files around on a regular basis, or transferring giant blobs of text or binary data, or looking for a storage format with metadata then csvs are an absolutely awful choice, but as with any problem you pick the right tool for the job, and csvs are a great tool when you want to easily move some structured alphanumeric data around in a format that's trivial to parse in any language and widely supported.

>it can be trivially generalized to use any character as a delimiter

But then you have the additional problem of trying to work out what the delimiter is. You can take an educated guess, but this isn't going to be 100% reliable.

This comes back to "picking the right tool for the job". If you're ingesting csvs/tsvs, you're expecting a given format anyway. If you expect the wrong delimiter, the worst thing that happens is you detect that the file has the wrong number of columns and fail out. Even if you're using a more structured format, if you ingest a file containing the wrong dataset, your process will still fail because the correct fields aren't there. No matter what your format is, you can't recover from somebody sending you the wrong file.

Make it a standard to always put the delimiter as the last character on the first line.