You don't even need to go to Unicode. ASCII has

  0x1C  FS  File Separator
  0x1D  GS  Group Separator
  0x1E  RS  Record Separator
  0x1F  US  Unit Separator

You just need a font with glyphs for them.

https://en.wikipedia.org/wiki/Delimiter#ASCII_delimited_text

Where can I find those on my keyboard? Entering escape sequences by hand isn't user friendly, if it works at all. One of the benefits of csv is that it's universal, I can open it in vim or write an awk script to extract values, someone else can open in notepad++, someone else should be able to open in excel.

Just thinking out loud here, but isn't part of the point is that they are not keyboard characters. If they're on the keyboard, then they will pop up in ordinary text, similar to the | (pipe) character and friends.

They are ^\ ^] ^^ and ^_, respectively. Of course, most text editors will interpret those keys as something else. In emacs you can enter them literally by using C-q (quote) first.

Since we're kind of talking about things from an I18N perspective, those characters are written like this on a bog standard Swedish keyboard:

    \ is AltGr+?
    ] is AltGr+9
    ^ is Shift+^, then space
    _ is Shift+-

AltGr is the right Alt key, to the right of the space bar.

So none of those are single keys, which means that combining them with Ctrl to write control characters becomes almost comically difficult. Not very accessible to typical users, I'd say.

Actually \ is AltGr++, i.e. "+" is the symbol you get from that key without any modifier. With shift you get ? and with AltGr you get \. My bad, and too late to edit.

Herein lies the slippery slope to APL...