News for hackers. Stuff that matters.
Re:Libgrapheme: A simple freestanding C99 library for Unicode by duskwuff on Tuesday November 15, 2022 @05:57PM (#33616375)
> First, nobody actually seems to know how bidirectional text should interact with terminal control sequences...

This goes beyond just bidirectional text. The traditional behavior of text in a terminal is based around two key assumptions, both of which break down catastrophically when dealing with non-ASCII text:

1) The state of a terminal can be represented as a set of cells, each of which has exactly one glyph in it and can be drawn independently from any other cell.

2) Printing a character will write a glyph to the cell the cursor is in and move the cursor to the right by one cell (or down to the next line).

The first assumption breaks down when dealing with full-width characters and ligatures/complex scripts, but can at least be papered over to handle full-width. The second assumption breaks down when exposed to virtually any interesting typographical feature (RTL, combining characters and ZWJ, shaped characters, etc). And I'm not sure it's possible to fix without some pretty substantial changes to how terminals operate -- standard terminal control sequences, and the code that uses them, are all built around these assumptions; introducing new behaviors like "the cursor doesn't always move from left to right" or "erasing the middle of a string might change how the rest of it displays" will break existing applications.

The ECMA standards are of absolutely no help in the matter. They were written in the early 1990s, before Unicode came onto the scene. Their idea of "international language support" was supporting both French and German.

Re:Libgrapheme: A simple freestanding C99 library for Unicode by Dylan16807 on Wednesday November 16, 2022 @01:57AM (#33619822)
Combining characters don't usually break those rules. They just mean that a single glyph is no longer 1-4 bytes, it's 1-128 or unlimited.
Re:Libgrapheme: A simple freestanding C99 library for Unicode by duskwuff on Wednesday November 16, 2022 @02:14AM (#33619913)
Consider a program that prints "A", waits a second, then prints a combining mark. Before the combining mark shows up, the terminal has no way of knowing to expect one, so it'll render an unaccented "A"; it then has to go back and modify that glyph (breaking the second rule) when the combining mark appears.

A terminal can't defer rendering that "A" until another character shows up -- that'd break common use cases like echoing user input -- and it can't refuse to apply the combining mark just because it arrived later, because the input to a terminal is a stream of bytes, not characters, and that stream can be delayed or interrupted in any number of ways.

And that's without getting into other interactions with terminal features, like: what happens if you move the cursor, or change its attributes, between the base character and its combining mark? How do you apply a combining mark to the last character on a line, since printing that character moves the cursor to the next line? How do combining marks interact with backspace? And so on.

Re:Libgrapheme: A simple freestanding C99 library for Unicode by Dylan16807 on Wednesday November 16, 2022 @07:28AM (#33622006)
I wouldn't have said that modifying the last-printed glyph breaks rule 2. Either way, I think modifying the glyph is the better answer.

If you send a console command to move the cursor or do anything else to it, the simple answer is to make that a barrier between glyphs. If you output a combining character after doing that, the combining character goes into the next cell all by itself.

Backspace sent to a terminal should move the cursor left by one cell as usual. A backspace sent by a user would probably delete one glyph if no IMEs are involved. If an IME is involved that's a different problem than plain old combining characters.

Being the last character on a line isn't really any different from being in the middle of a line. The cursor auto-moves, but if the next bytes the terminal gets are combining characters then you can pretty easily shove them into the last-printed glyph.

Re:Libgrapheme: A simple freestanding C99 library for Unicode by duskwuff on Wednesday November 16, 2022 @04:02PM (#33630532)
> If you send a console command to move the cursor or do anything else to it, the simple answer is to make that a barrier between glyphs. If you output a combining character after doing that, the combining character goes into the next cell all by itself.

Maybe. I'd be concerned about what effects that might have on applications which need to be able to display multiple streams of output at once, like tmux.

> Being the last character on a line isn't really any different from being in the middle of a line

You might be surprised. After printing the last character on a line, the cursor goes into a weird "wrapnext" state, where the cursor stays at the end of the line, but will wrap to the next line (potentially triggering scrolling) when the next character is output. This state is highly ephemeral; it can't be entered through control sequences, even by DECSC/DECRC.