News for hackers. Stuff that matters.
Re:How does apt render its fancy progress bar? by gildas on Wednesday October 13, 2021 @06:15AM (#28850360)
For those who wonder how these "\033[1A..." notations are specified, look up "ANSI escape code" on the net, e.g. https://en.wikipedia.org/wiki/ANSI_escape_code
Re:How does apt render its fancy progress bar? by p-e-w on Wednesday October 13, 2021 @07:33AM (#28850906)
And if you want to play around with escape sequences from your shell, you can do so using `echo -e "\e[1A..."` in most shells (the `-e` flag enables backslash interpretation, while `\e` stands for the ESC character, alias `\033`).

Note also that (traditional) escape sequences cannot execute arbitrary code, so you can't really break anything that closing and reopening the terminal won't fix.

Re:How does apt render its fancy progress bar? by MayeulC on Wednesday October 13, 2021 @10:23AM (#28852460)
> you can't really break anything that closing and reopening the terminal won't fix

Or typing `reset`, which is sometimes also necessary after accidentally piping binary to stdout.

Re:How does apt render its fancy progress bar? by p-e-w on Wednesday October 13, 2021 @10:44AM (#28852775)
`reset` won't work if you have used the escape sequence that disables raw mode, at which point the shell won't be able to process keyboard input anymore and you simply cannot issue commands. I'm not aware of any method for fixing this that doesn't involve restarting the terminal.
Re:How does apt render its fancy progress bar? by remram on Wednesday October 13, 2021 @12:01PM (#28853881)
You should still be able to `reset` in line-buffering mode.
Re:How does apt render its fancy progress bar? by anothermindbomb on Wednesday October 13, 2021 @01:23PM (#28854908)
`stty sane` not work?
Re:How does apt render its fancy progress bar? by withinboredom on Wednesday October 13, 2021 @01:01PM (#28854646)
And if you really want to have fine, put them in your git commits and stand back as `git log` destroys your coworkers terminal.
Re:How does apt render its fancy progress bar? by quotemstr on Wednesday October 13, 2021 @09:42AM (#28851936)
You can be sloppy and hardcode escape sequences in your shell script, or you do it right and use the tput program to look up the right sequences at runtime.
Re:How does apt render its fancy progress bar? by p-e-w on Wednesday October 13, 2021 @10:36AM (#28852642)
ANSI escape sequences are a standard. Relying on that standard isn't sloppy, any more than relying on any other standard. Terminfo is today basically a hack to work around the problem that some uncommon sequences differ between terminals, but others (like cursor movement and SGR, which cover 99% of use cases) have been well-established for 30 years. I much prefer using such sequences directly over the complexity of shelling out to an external program. I'm not going to pretend that there is a chance my script might have to run on some obscure text terminal from the 80s some day.
Re:How does apt render its fancy progress bar? by rbonvall on Wednesday October 13, 2021 @10:51AM (#28852883)
Standardness notwithstanding, using tput is more readable and memorable. I can type this from memory:

    echo "I'm $(tput setaf 4)blue$(tput sgr0)."
but I couldn't get this one right without looking it up:

    echo -e "I'm \033[0;34mblue\033[0m."
Re:How does apt render its fancy progress bar? by junon on Wednesday October 13, 2021 @12:50PM (#28854505)
Because you're using a lot more characters than you need.

    echo -e "I'm \e[34mblue\e[m."
However, `tput` is definitely the more 'correct' approach. What actual benefit it affords you over the hardcoded escapes is left to edge-cases and very, very old or otherwise niche terminal emulators.

By the way, the `3x` means foreground, the `4x` means background. `x` is a number between 0 and 7 (inclusive) that indicates the color. `x=8` means "extended"/non-standard and generally takes a few more control codes afterward, and `x=9` means "reset".

If you know binary, you can remember the 0-7 colors. It's a 3-bit code corresponding to BGR (where B is the MSB).

    BGR
    000 = 0 = black
    001 = 1 = red
    010 = 2 = green
    011 = 3 = yellow
    100 = 4 = blue
    101 = 5 = magenta
    110 = 6 = cyan
    111 = 7 = white
You can also add 60 to the code to make it "bright", e.g. red foreground (31) can be made bright red by adding 31+60=91. Same can be applied to backgrounds (4x+60=10x).

The bright codes are less supported though admittedly I've never seen a modern emulator that doesn't. It also gives you bright colors on old Windows cmd.exe prompts without needing the bold mode (1).

Re:How does apt render its fancy progress bar? by faho on Wednesday October 13, 2021 @11:46AM (#28853683)
Still not particularly readable or memorable. You still need to remember that blue is color number 4 and that "sgr0" is "exit_attribute_mode", what you might call "reset".
Re:How does apt render its fancy progress bar? by tomc1985 on Wednesday October 13, 2021 @12:26PM (#28854217)
Store the sequences in variables and put those instead. You get readability and locality!
Re:How does apt render its fancy progress bar? by nick__m on Wednesday October 13, 2021 @12:51PM (#28854519)
I was about to post the exact same reply so I will post it as an example:

  #!/bin/sh
  BLUE=$(tput setaf 4)
  RST=$(tput sgr0)
  echo "I'm ${BLUE}blue${RST}."
Re:How does apt render its fancy progress bar? by mmphosis on Wednesday October 13, 2021 @05:34PM (#28857996)

  alias red='echo -ne "\e[31m"'
  alias yellow='echo -ne "\e[33m"'
  alias green='echo -ne "\e[32m"'
  alias blue='echo -ne "\e[34m"'
  alias cyan='echo -ne "\e[36m"'
  alias violet='echo -ne "\e[35m"'
  alias grey='echo -ne "\e[90m"'
  alias gray='echo -ne "\e[90m"'
  alias white='echo -ne "\e[37m"'
  alias bold='echo -ne "\e[1m"'
  alias flash='echo -ne "\e[7m\e[5m"'
  alias normal='echo -ne "\e(B\e[m"'
Re:How does apt render its fancy progress bar? by not2b on Wednesday October 13, 2021 @12:07PM (#28853999)
Back when termcap and later terminfo were developed, most terminals didn't support ANSI escape sequences, at all, and used completely different control codes. But that was 30+ years ago. It was for actual dumb terminals that could only display text. These days, it's a safe bet that any terminal emulator will support the ANSI escape sequences.
Re:How does apt render its fancy progress bar? by spudlyo on Wednesday October 13, 2021 @02:46PM (#28855833)
Is it though? What if I have my TERM set to 'dumb' because I'm running your CLI command inside a very limited console embedded inside a text editor or other tool? At the very least you should `test -t` in your script to ensure STDOUT isn't being redirected to a file so you don't fill it with garbage escape sequences.
Re:How does apt render its fancy progress bar? by junon on Wednesday October 13, 2021 @12:49PM (#28854494)
> ANSI escape sequences are a standard.

Hi, Chalk maintainer here. They are most certainly not standardized, despite frequently shown with the "ANSI" nickname - at least, not by any widely accepted standard I'm aware of.

The only standardization I could ever find myself was that of a document format similar to PostScript, but it didn't describe the codes that XTerm originally adopted. You would have to ask the original implementors how/why they chose the codes that they did.

Serial/teletype (TTY) escape codes are one of the most archaic, still-widely-used functions of terminal emulators and as such sprung up before the dawn of widely agreed-upon standards. There are thousands of them, not just those for rendering, and they were historically used for serial transmission control for e.g. printers and other UART devices.

Thus, many (most, actually) escape codes have nothing to do with rendering and instead control the behavior of the two nodes at either end of a serial line.

Since most of these devices used proprietary codes - including those for rendering - a common subset of those related specifically to terminal emulation/TTYs were derived and a database of those codes was formed. This database is usually installed on *nix installations via the `ncurses` package and is generally referred to as the "terminfo database" or simply "terminfo" or "termdb".

Each file in the (very, very extensive) database is a binary file format that includes a complete stack-based scripting language for string-based escape codes (there are a few datatypes supported in the file format, some of which are purely to indicate capabilities and whatnot).

I wrote an implementation of a Terminfo parser with scripting language support in a C++ library if anyone is interested. I'd have to dig it up if someone would like to use it - just let me know.

The entry for the given terminal session is selected via the `TERM` environment variable in most cases, which ncurses-based applications (and other applications that support terminfo) reads in order to select and parse the correct terminfo entry.

Most of the common escapes we use today stem from the XTerm set of codes - namely, `TERM=xterm-256color` - since supporting Terminfo is a beast and can hurt performance when properly supported (since each 'invocation' of the escape is really an evaluation of a small script).

Since XTerm made the codes simple to render out (CSI(\x1b) <mode>[;<mode>[;...]] m), and since most terminal emulators chose to model their escapes off of XTerm, many application developers started to hardcode the XTerm escapes directly in lieu of using Terminfo. This was further set in stone when the often-referenced Wikipedia page[0] for ANSI escapes was written, claiming that those were the only codes - which is very far from the case.

XTerm in particular re-uses a lot of codes from popular-now-archiaic terminals such as the VT100, which introduced bold types among other things (but not color!). I believe it was the VT520 or something like that that had the first semblance of color codes, but my memory is fuzzy and I'm not in a position to research it right now. I'll leave that as an exercise to the reader.

For the most part this has worked out okay. Naive/toy emulators recognize these codes, some shells (not many, but some) actually transform the escapes, some libraries translate them to Windows API calls (e.g. libuv, the I/O and event loop behind Node.js), and most established terminal emulators choose to use the xterm-256color by default anyway.

There have been a number of closed-door discussions between terminal emulator vendors about modernizing all of this but, incredibly, PowerShell is the only effort I've seen widely adopted or used that has made strides in improving this whole debacle.

I'm sure there are some errors in what I wrote - feel free to correct me if I'm wrong. But this is my understanding of the messy world of "ANSI" escape characters.

[0] https://en.wikipedia.org/wiki/ANSI_escape_code

Re:How does apt render its fancy progress bar? by arp242 on Wednesday October 13, 2021 @07:08PM (#28858680)
The standard is ECMA-48; for example this 1991 version[1] lists all the SGR codes to colour stuff (page 61), the cursor movement codes, etc. The 1991 is just the first version that came up, I believe some of this goes back to the late 70s.

Terminfo goes back to the early 80s, during the time of the Great Unix Wars when everyone was doing everything different for the fun and sake of it. It contains a lot of cruft. People are mentioning ADM-3A terminals here and that's all very nice, but these are machines from the 70s. No one is using them, except some people for the fun/historical value. Do people write software to be compatible with Bell Labs Unix from 1976 or 1BSD that Bill Joy was working on? Of course not.

AFAIK there are very few modern terminals (or rather, terminal emulators) that don't support that basic set of escape codes. I can't really find any in a quick glance at my system's terminfo database.

For basic operations it's pretty safe to rely on, at least if you care about Unix only (not entirely sure about Windows, I believe it's all different there). Once you go beyond that it gets a bit more iffy.

There are loads of programs that hard-code these things, including very popular once. The issue trackers of these projects are not getting filled with people reporting garbled output.

[1]: https://www.ecma-international.org/wp-content/uploads/ECMA-4...

Re:How does apt render its fancy progress bar? by junon on Thursday October 14, 2021 @04:21AM (#28861745)
ECMA-48 was what I was referring to, thanks. And no, I'm not convinced it was the standard many were conforming to. Perhaps Xterm, but not everyone.

Further ECMA and ANSI are two separate standards bodies. While ECMA-48 is listed as one of the standards on the Wikipedia page I've yet to find anything that claims to conform to that standard. The same with the other listed "standards".

Re:How does apt render its fancy progress bar? by arp242 on Thursday October 14, 2021 @07:05AM (#28862740)
Actually that Wikipedia page lists the history of standards:

"The ANSI standard attempted to address these problems by making a command set that all terminals would use and requiring all numeric information to be transmitted as ASCII numbers. The first standard in the series was ECMA-48, adopted in 1976. It was a continuation of a series of character coding standards, the first one being ECMA-6 from 1965, a 7-bit standard from which ISO 646 originates. The name "ANSI escape sequence" dates from 1979 when ANSI adopted ANSI X3.64. The ANSI X3L2 committee collaborated with the ECMA committee TC 1 to produce nearly identical standards. These two standards were merged into an international standard, ISO 6429. In 1994, ANSI withdrew its standard in favor of the international standard."

So ANSI X3.64 was the original, but things has since converged and it has been superseded by the ECMA/ISO one.

The document that many people use is probably the Xterm ctlseqs page, which is just the same as the ECMA codes with additional notes, extensions, etc. So I suppose that's the de-facto standard now.

Re:How does apt render its fancy progress bar? by chasil on Wednesday October 13, 2021 @04:43PM (#28857440)
Below is the original manual for a Lear-Siegler ADM-3A terminal, which did not use any of the VT100-derrived escape codes (and was much less expensive at the time).

The ADM-3A is notable as it was used by Bill Joy to develop the "vi" editor, and the key layout should be familiar.

http://www.bitsavers.org/www.computer.museum.uq.edu.au/pdf/D...

Re:How does apt render its fancy progress bar? by quotemstr on Wednesday October 13, 2021 @10:39AM (#28852703)
OpenGL is a standard too, but an OpenGL program will break on a system supporting only Vulkan. Terminfo exists for a reason and that reason is still there today. For example, I can run under TERM=dumb to disable fancy output for various reasons.

No, terminfo isn't a hack, ANSI escapes aren't ubiquitous, and every single time I see someone hardcode escape sequences in some script, my opinion of that person declines.

Re:How does apt render its fancy progress bar? by klodolph on Wednesday October 13, 2021 @10:43AM (#28852747)
Imagine living in a world where OpenGL has been around since the 1970s, stable for a long time, and where there’s no sign of Vulkan on the horizon.

That’s the world of terminals. There’s no sign that the way terminals work is going to change. These standard escape sequences are old.

Re:How does apt render its fancy progress bar? by fiddlerwoaroof on Wednesday October 13, 2021 @11:42AM (#28853617)
Until your program is run in a emacs window (lint-staged’s progress bars mess up magit’s output buffer, making it really difficult to figure out what the problem is) or is piped through grep.

tput will do the right thing because of terminfo, hand-coded escape sequences will make someone curse you.

Re:How does apt render its fancy progress bar? by froh on Wednesday October 13, 2021 @08:04AM (#28851086)
To make things portable look up termcap(5) and tput(1) to ask the termcap database for the right control sequence for your terminal, and use those.
Re:How does apt render its fancy progress bar? by julienpalard on Wednesday October 13, 2021 @10:18AM (#28852394)
Interestingly apt don't do this, they hardcode sequences as I did.

Maybe they bet those sequences are the same on all terminals since 1970, espacially those marked "DEC Private" and it won't change soon?

I honestly don't know.

Re:How does apt render its fancy progress bar? by froh on Thursday October 14, 2021 @08:03AM (#28863163)
Most terminal emulations follow the ANSI terminal https://en.m.wikipedia.org/wiki/ANSI_escape_code , so these hard coded escape sequences simply happen to work on most virtual terminals, including windows console (running CMD) and windows terminal.
Re:How does apt render its fancy progress bar? by penguin_booze on Wednesday October 13, 2021 @12:11PM (#28854056)
This in-band control signalling was mind bending the first time I learned about it. "How come other programs print text in colour, but mine is only plain? Wow - I can print special sequences and that controls color?! This means the console must be interpreting what I'm sending, not doing a simple lookup and display!".
Re:How does apt render its fancy progress bar? by katzeilla on Wednesday October 13, 2021 @07:48AM (#28850991)
... and in case you messed up your console, typing `reset` followed by enter will fix it.
Re:How does apt render its fancy progress bar? by julienpalard on Wednesday October 13, 2021 @10:21AM (#28852429)
I had to do it quite a few time when implementing this one ;)

I often type "C-c" or "enter" before typing reset to ensure I have nothing before it in the buffer.

A bit like I often do "Enter Enter ~ .` instead of just `~.` to kill a dying SSH session (I don't know why I hit enter twice here... a single time is enough.) to ensure the ~ is at the beginning of the buffer (if it's not, it won't work).

Re:How does apt render its fancy progress bar? by faho on Wednesday October 13, 2021 @07:59AM (#28851058)
Or, if you managed to mess up the terminal modes and enter doesn't work, try pressing ctrl-j instead.