News for hackers. Stuff that matters.
Re:ASCII Delimited Text – Not CSV or TAB delimited text by scelerat on Wednesday March 26, 2014 @02:02PM (#7475124)
ASCII 0-31 are called "control" characters so it should come as no surprise that you could type them using the Control key.

Unit Separator is Control-_ (underscore) and Record Separator is Control-^ (caret), for instance.

Most modern text editors won't pass through every control character. Vim lets me type the unit separator, but not the record separator, for instance.

Control-C and Control-D, End of Text and End of Transmission, still have utility in most shells, and it goes back directly to these ASCII control characters.

https://www.cs.tut.fi/~jkorpela/chars/c0.html

Re:ASCII Delimited Text – Not CSV or TAB delimited text by thwarted on Wednesday March 26, 2014 @02:38PM (#7475401)
Unit Separator is Control-_ (underscore) and Record Separator is Control-^ (caret), for instance.

Most modern text editors won't pass through every control character. Vim lets me type the unit separator, but not the record separator, for instance.

Vim has bindings for some control key combos, which is why you can't type them directly. I can type control-_ without issue, but to get an ASCII 30 (RS) to show up, I have to type control-v first (just like when at the shell prompt).

Useful hint: if you look at the output of man ascii (at least on linux with man-pages-3.22) find the control character you want to type in the left column and look in the same row on in the right column to find the letter/key to use with the control key.

For example, to type NAK, it's <control-u>. Vertical tab is <control-k>:

   $ echo <control-v><control-k> | od -c
   0000000  \v  \n
This is useful for control characters that don't have backslash escape expansions.
Re:ASCII Delimited Text – Not CSV or TAB delimited text by raldi on Wednesday March 26, 2014 @02:44PM (#7475457)
...and that's why, when running a command that's reading what you're typing to stdin, you press Ctrl-D to tell it that you're done. Because, as your "man ascii" trick shows, this generates the "end of transmission" character.

Similarly, Ctrl-L clears the screen in most Unix apps because that generates the "form feed" character, and on a line printer or paper-based teletype terminal, "form feed" means to advance to the next page -- a nice empty clear piece of paper.

Re:ASCII Delimited Text – Not CSV or TAB delimited text by thwarted on Wednesday March 26, 2014 @04:13PM (#7476259)
when running a command that's reading what you're typing to stdin, you press Ctrl-D to tell it that you're done. Because, as your "man ascii" trick shows, this generates the "end of transmission" character.

This is slightly different than being able to actually generate those characters as input. If you put a control-d in a file, nothing will stop reading the file when it sees the control-d, it's just another byte. readline knows how to interpret a bunch of control characters. There's also the terminal driver that has interpretations, which you can see with `stty -a`. When the tty layer sees a control-d or a control-c, it closes stdin or generates SIGINT, respectively. But even this can be dependent on if you're on a pty or attached directly to a serial line. There's nothing special about the mapping of control-d to end-of-transmission character, that's just how the layer that sees it is interpreting it. You can, for example, change how control-c is interpreted by `stty intr ^B`, which will make control-b generate SIGINT. And there's a way to put the terminal driver in complete transparent/passthru mode.

Re:ASCII Delimited Text – Not CSV or TAB delimited text by raldi on Wednesday March 26, 2014 @04:55PM (#7476592)
Sure, the default is overridable, and the character has no magic effect when found in a binary file, but there's a reason Ctrl-D was chosen as the standard keystroke sequence to end a stdin transaction, and it has everything to do with the fact that "D" shows up next to "end of transmission" in `man ascii`.
Re:ASCII Delimited Text – Not CSV or TAB delimited text by thwarted on Wednesday March 26, 2014 @10:57PM (#7478297)
Thank you for clarifying something that no one was disputing or event mentioned.
Re:ASCII Delimited Text – Not CSV or TAB delimited text by qu4z-2 on Wednesday March 26, 2014 @11:44PM (#7478470)
Yeah, I hate it when people point out interesting things I didn't know before too.
Re:ASCII Delimited Text – Not CSV or TAB delimited text by thwarted on Thursday March 27, 2014 @03:21AM (#7479046)
I'm sure raldi can repeat, again, making it 3 times, that control-d means end of transmission and that's why it was chosen for signalling end of stdin, on a thread about how to type the control characters.
Re:ASCII Delimited Text – Not CSV or TAB delimited text by oneeyedpigeon on Wednesday March 26, 2014 @09:14PM (#7477900)
I am going to love not having to type out 'clear' all the time - thank you!
Re:ASCII Delimited Text – Not CSV or TAB delimited text by sopooneo on Wednesday March 26, 2014 @03:33PM (#7475887)
what about Ctrl-C to break out of stuff? Does that correspond to an ascii control character?
Re:ASCII Delimited Text – Not CSV or TAB delimited text by elwin on Wednesday March 26, 2014 @04:06PM (#7476203)
Ctrl-C stops the program because the TTY driver watches for it and sends an interrupt signal when it appears. It's a slightly different mechanism.

There are more details in "man 3 termios", but be warned, the TTY layer is not a pleasant subject for reading.

Re:ASCII Delimited Text – Not CSV or TAB delimited text by danielweber on Wednesday March 26, 2014 @04:52PM (#7476566)
I was disappointed that Ctrl-S and Ctrl-Q don't correspond (at least on the first ASCII chart I googled) to anything, because Ctrl-S stops text on my Unix terminals the exact same way it did on my brother's Apple ][e in 1984.
Re:ASCII Delimited Text – Not CSV or TAB delimited text by jrochkind1 on Wednesday March 26, 2014 @05:13PM (#7476715)
> I was disappointed that Ctrl-S and Ctrl-Q don't correspond (at least on the first ASCII chart I googled) to anything

I'm pretty sure every possible combination of 7-bits is assigned SOME name in ascii, and Ctrl-any-case-insensitive-letter is a defined 7bit value.

This chart says control-s is `DC3 (Device Control, X-OFF)` and control-q is `DC1 (Device Control, X-ON)`. Yup, that's what they do alright.

http://www.unix-manuals.com/refs/misc/ascii-table.html

Don't forget everyone's favorite control-G, BEL, useful sending beeps across the wire on old school IRC, BBSen, and chat services. Or, you know, telegraph machines or something.

Re:ASCII Delimited Text – Not CSV or TAB delimited text by csixty4 on Wednesday March 26, 2014 @05:40PM (#7476866)
For anyone wondering, X-ON and X-OFF are flow control characters. A device sends X-OFF to say "my buffer is full, stop sending data" and X-ON when it's ready to resume.

On most Unix terminals, these have the effect of pausing and resuming the display when a bunch of data is scrolling by.

Re:ASCII Delimited Text – Not CSV or TAB delimited text by __del__ on Wednesday March 26, 2014 @07:03PM (#7477301)
...and they're helpfully disabled with stty -ixon
Re:ASCII Delimited Text – Not CSV or TAB delimited text by danielweber on Wednesday March 26, 2014 @07:19PM (#7477390)
Oh, thank you. The reference I found[1] only showed them as "device control." I thought I imagined them having that.

Also, the classic control-G is still in C. "\a" is "alert (beep)."

[1] http://www.asciitable.com/

Re:ASCII Delimited Text – Not CSV or TAB delimited text by fineline on Wednesday March 26, 2014 @06:57PM (#7477260)
I think you mean your brother's Apple //e - they switched from the original ][ to // for later models. That's what I started programming on too - except in my case it was my Dad's.
Re:ASCII Delimited Text – Not CSV or TAB delimited text by danielweber on Wednesday March 26, 2014 @07:20PM (#7477401)
Ah, but the boot screen said "APPLE ]["
Re:ASCII Delimited Text – Not CSV or TAB delimited text by jloughry on Wednesday March 26, 2014 @04:19PM (#7476296)
Ctrl-C was originally "End of Text".
Re:ASCII Delimited Text – Not CSV or TAB delimited text by dhimes on Wednesday March 26, 2014 @06:10PM (#7477010)
That's a terrific hint. So the ctrl-<dec-val + 64 char> = <dec val char>
Re:ASCII Delimited Text – Not CSV or TAB delimited text by jrochkind1 on Wednesday March 26, 2014 @05:08PM (#7476678)
> Unit Separator is Control-_ (underscore) and Record Separator is Control-^ (caret), for instance.

And this is in fact how they show up in vim (or at least my fairly uncustomized vim), using ^ to stand for ctrl as was once conventional: `^_` and `^^`

The legacy MARC binary format still used for library data uses ascii 29, 30, and 31 -- although for reasons with probably some bizarre historical definition uses them DIFFERENTLY than defined in ascii.

0x1D == 29 == ascii group seperator == MARC Record separator

0x1E == 30 == ascii record seperator == MARC Field Terminator

0x1F == 31 == ascii unit separator == MARC subfield seperator

http://en.wikipedia.org/wiki/MARC_standards

Re:ASCII Delimited Text – Not CSV or TAB delimited text by pjbrunet on Wednesday March 26, 2014 @05:24PM (#7476777)
Back in the day with a numeric keypad you could type Alt-(number) to get any ASCII character, but not sure if that still works. In Firefox apparently Alt-1 takes you to the first tab, Alt-2 second tab...
Re:ASCII Delimited Text – Not CSV or TAB delimited text by BorgHunter on Wednesday March 26, 2014 @07:20PM (#7477400)
In most graphical *nix environments that I'm familiar with, if you hold Control and Shift¹ and type U, you can type any Unicode character by its (hex) code point. For instance, if you want an em dash, which is U+2014, hold Control and Shift, and type "U2014".

¹You can release Control and Shift after typing "U", in which case the character will appear after you type a space. Or, you can hold Control and Shift while typing the code point, in which case the character will appear after you release either modifier key.

Re:ASCII Delimited Text – Not CSV or TAB delimited text by pjbrunet on Friday March 28, 2014 @01:54PM (#7488854)
I just tried in Debian, works! Very handy.
Re:ASCII Delimited Text – Not CSV or TAB delimited text by chinpokomon on Wednesday March 26, 2014 @10:31PM (#7478221)
On Windows yes, just proceed the number with a 0 to get Unicode. See the "How to enter Unicode characters in Microsoft Windows" page [1] for more details, including one of my favorite tricks, using Alt-X in WordPad. It is especially useful when you want to know what the code point is for a Unicode character.

[1] http://www.fileformat.info/tip/microsoft/enter_unicode.htm

Re:ASCII Delimited Text – Not CSV or TAB delimited text by T-hawk on Thursday March 27, 2014 @12:00PM (#7481172)
Turn on Num Lock and it still works, including in Firefox.
Re:ASCII Delimited Text – Not CSV or TAB delimited text by Terretta on Wednesday March 26, 2014 @07:15PM (#7477359)
Alt-(3 digits)
Re:ASCII Delimited Text – Not CSV or TAB delimited text by npongratz on Wednesday March 26, 2014 @02:36PM (#7475379)
Vim will let you type the record separator, but you may have to precede it with Ctrl-V. In other words, type the Ctrl-V Ctrl-^ key sequence.

At least, that worked for me. Interesting that the unit separator does not have the same requirement to precede it with Ctrl-V.

More info in the vim docs:

http://vimdoc.sourceforge.net/htmldoc/insert.html#ins-specia...

Re:ASCII Delimited Text – Not CSV or TAB delimited text by groovy2shoes on Wednesday March 26, 2014 @04:46PM (#7476518)
Yes, Ctrl-V in {insert,command} mode lets you input literal control characters, as well as numeric codes (31 for unit separator, 30 for record separator). If you're using vim on Windows, however, note that the default vimrc loads mswin.vim which changes Ctrl-V to paste. In that case, you can simply remove mswin.vim from your vimrc or use Ctrl-Q instead. (See :h i_Ctrl-V).

You can also use Ctrl-K to enter RFC1345 digraphs, in which case unit separator is 'Ctrl-K US' and record separator is 'Ctrl-K RS'. (See :h Ctrl-K).

Emacs has similar features, as well as a nifty TeX input mode: http://stackoverflow.com/q/6269618

Re:ASCII Delimited Text – Not CSV or TAB delimited text by Pxtl on Wednesday March 26, 2014 @02:08PM (#7475172)
Huh. I wonder how many of those are completely disused, or would be meaningless in a file?