News for hackers. Stuff that matters.
Re:Terminal Emulators Battle Royale – Unicode Edition by zokier on Saturday December 23, 2023 @07:40AM (#38743820)
The key observation to me here is that the character cell grid abstraction that terminals are built upon is poor fit for modern general-purpose multi-lingual text rendering. Works well enough with plain ASCII but anything beyond that gets increasingly messy
Re:Terminal Emulators Battle Royale – Unicode Edition by lor_louis on Saturday December 23, 2023 @08:55AM (#38744243)
I would argue that strings as an array of individually addressable 'characters' is a bad fit for modern multilingual text handling.
Re:Terminal Emulators Battle Royale – Unicode Edition by _a_a_a_ on Saturday December 23, 2023 @10:22AM (#38744818)
Why? How does it fail?
Re:Terminal Emulators Battle Royale – Unicode Edition by lor_louis on Sunday December 24, 2023 @10:41AM (#38754197)
In utf-8, bytes (uint8_t) may not represent a whole "code point". A code point being an individually meaningful element in utf-8 like a space an 'e' or a modifier code point like an accent or a ZWJ. Most utf-8 libraries will let you address individual code points but it might still garble the text if you split between an 'e' and a '`'. To prevent this, splitting should be done in between graphemes (sequences of code points that render like a single unit*). And even graphemes have their problems.

Very interesting blog post about graphemes that parallels my experience writing a terminal text editor: https://mitchellh.com/writing/grapheme-clusters-in-terminals

* Event that is not a proper description of a grapheme

And don't even get me started on regexes

Re:Terminal Emulators Battle Royale – Unicode Edition by _a_a_a_ on Sunday December 24, 2023 @11:03AM (#38754353)
Yes, I understand a little about Unicode in this kind of problem, but a code point is an individual logical item even if it is composed of multiple bytes; being a kind of 'string' in itself. I should have asked more carefully, what would be a better system in your view?

Thanks for the link, will check it out after Christmas.

Re:Terminal Emulators Battle Royale – Unicode Edition by lor_louis on Tuesday December 26, 2023 @07:44PM (#38777730)
I personally believe that Swift's strings where graphemes are the smallest indexable unit are the gold standard for writing logic that might truncate multilingual text. It's still not perfect though, they add overhead and updates to Unicode might change behaviour so there's that but it should handle most cases gracefully.
Re:Terminal Emulators Battle Royale – Unicode Edition by eviks on Saturday December 23, 2023 @09:00AM (#38744264)
It fails miserably even with ASCII, works only with a monospace variety of it, and non-monospace isn't modern...
Re:Terminal Emulators Battle Royale – Unicode Edition by bee_rider on Saturday December 23, 2023 @04:17PM (#38748328)
That’s an interesting thought, IMO. These flexible width fonts are often thought of as localization-assistance features, to help fit other languages in the terminal world. But English also had to be modified to fit into fixed-width fonts. I wonder how the discussion would change if this technology had been popularized originally in another country.
Re:Terminal Emulators Battle Royale – Unicode Edition by Per_Bothner on Saturday December 23, 2023 @10:10AM (#38744735)
I am working on a re-write (https://github.com/xtermjs/xterm.js/issues/4800) of the xterm.js "BufferLine" data structures that (among other benefits) could potentially support variable-width fonts in the terminal. Instead of a 2-dimensional grid of character cells, the primary data structure will be a list of logical lines, each consisting of characters and attributes.

A rather modest enhancement to the vt100/xterm escape sequences would be a mode where screen addressing would be in terms of logical characters and logical lines. The application should not need to know the width of characters or how lines are broken into screen rows. This makes use of variable-width fonts practical - which is necessary for decent rendering of many scripts.

DomTerm's display engine (https://domterm.org) does support lines containing a mix of monospace and variable width characters, and know how to wrap such lines. I'm working on long-term replacing DomTerm's native display engine with xterm.js (for speed and other reasons) which requires making xterm.js more versatile.