Also, ReGIS and Tektronix (disclaimer: I proposed including Tek graphics in VTE) could also be very useful.

Isn't this can of worms inherent to the subject, anyway? This question has been asked - and answered - by pretty much every rendering/display system, ever.

Seems to me, sixel adoption at this point is the pre-cursor to optimization ...

Recently discovered Kitty's graphics protocol (https://sw.kovidgoyal.net/kitty/graphics-protocol/) which has more features or at least more documented ones :)

DEC invented Sixels back in the ’80s, and they were serious about their docs, so the corresponding chapter of the VT3xx manual[1] is probably as good as it gets.

> I.e. how large are the pixels [...].

Historical implementations likely assume the relation between pixels and character cells that’s implied by the geometry of the DEC fonts. I’ve seen a lot of arguing about adapting this to the modern world, but I don’t know if a consensus has emerged.

[1] https://vt100.net/docs/vt3xx-gp/chapter14.html

  LF='\n'; ESC='\e'; CSI=$ESC'['; DCS=$ESC'P'; ST=$ESC'\\'
  CUF=$CSI'%dC' # cursor forward
  SIXEL=$DCS'q%s'$ST # sixel image
  printf "before $SIXEL$CUF after$LF" \ 
    '#0;2;0;0;0#1;2;100;100;0#2;2;0;100;0#1~~@@vv@@~~@@~~$#2??}}GG}}??}}??-#1!14@' \ 
    2

- in XTerm (xterm -xrm "XTerm*decTerminalID: vt340"), "before ", "HI", " after", that is to say exactly what you want, out of the box;

- in Foot, "before ", "HI", newline, some spaces, "after";

- in Contour, "before ", "HI", enough newlines to clear the screen* (?..), no spaces* (?!..), "after".

OK, sez I, let’s just save the cursor position (DECSC, ESC 7) before the image and restore it (DECRC, ESC 8) afterwards, then skip over it; that is,

  DECSC=$ESC'7'; DECRC=$ESC'8' # add to definitions
  printf "before $DECSC$SIXEL$DECRC$CUF after$LF" # change format string

So it seems to me like what you want should work by default, except it doesn’t.

It should be possible to instead just treat the whole thing as a graphical overlay (by computing or directly asking for the character cell size, as Kirill Panov rightly admonishes me is possible with XTWINOPS) without touching the cursor; that’s what the “sixel scrolling” setting (DECSDM) is supposed to do. Then you can just manually move the cursor forward however many positions after you’re done drawing.

Except apparently the DEC manual (the VT330/340 one above) and DEC hardware contradict each other as to which setting of DECSDM (set or reset) corresponds to which scrolling state (enabled or disabled), and XTerm has implemented it according to the manual not the VT3xx[1,2,3]—then most other emulators followed suit[4]—then XTerm switched to following the hardware[5,6] (unless you and that’s what I’m seeing on my machine right now. So now you need to check if you’re on XTerm ≥ 369 or not[7]. And also for other terminals’ versions, because apparently that’s a thing now[8,9].

Again, ouch.

P.S. DEC had an internal doc for how their terminals should operate (DEC STD 070) [10]. It does not document DECSDM at all.

[1] https://github.com/wez/wezterm/issues/217#issuecomment-86449...

[2] https://github.com/hackerb9/lsix/issues/41

[3] https://github.com/dankamongmen/notcurses/issues/1782

[4] https://github.com/arakiken/mlterm/pull/23

[5] https://invisible-island.net/xterm/xterm.log.html#xterm_369

[6] https://invisible-island.net/xterm/ctlseqs/ctlseqs.html#h3-T...

[7] https://github.com/dankamongmen/notcurses/commit/0918fa251e2... (the correct version cutoff is 369 not 359, the patch contains a now-fixed bug)

[8] https://github.com/dankamongmen/notcurses/issues/2204

[9] https://github.com/dankamongmen/notcurses/blob/master/src/li... (look for mentions of invertsixel or invert80)

[10] http://www.bitsavers.org/pdf/dec/standards/EL-SM070-00_DEC_S...

Nice. I wish I'd had that years ago when the maintainer of a then-popular virtual terminal got very angry at me for suggesting that DECCOLM (set 80/132 columns) should not change the number of lines.

    "\x1b[14t"

You just send the particular query (e.g. ‘CSI 14 t’) and the terminal sends back a response in the defined form¹. Of course you'll want raw mode, echo off, etc. Normally a library like curses does this for you. If you want to see, https://gist.github.com/kpschoedel/6a87ec2157ce2140be69193d1... (I just whipped this up to answer the question; don't expect production quality)

¹ https://invisible-island.net/xterm/ctlseqs/ctlseqs.html#h3-F...

It's still in the early phase of development, but actually fully functional.

> After I took over the maintainership of libsixel I unfortunately decided it cannot support the security demands of Kitty, it is too insecure internally. I need to write a Rust library or something.

[1] https://github.com/kovidgoyal/kitty/issues/2511

[2] https://github.com/kovidgoyal/kitty/issues/2511#issuecomment...

Sorry to be that guy, but what is a "NIH"? All I know is the https://www.nih.gov/ :)

I guess that may br true in some senses, but per Nick Black[1] (of Notcurses fame) the set of Kitty graphics protocol implementations consists of Kitty and Wezterm, that’s it.

[1] https://github.com/alacritty/alacritty/issues/910#issuecomme...

Having these intermixed between the regular scrollable textual output is quite powerful.

> If you’re working with Jupyter notebooks in a terminal only environment, like an SSH server or a container, or just prefer working in the terminal, then euporie is the tool for you!

Uses Kitty ("Currently only the kitty and WezTerm terminals support this"), Sixels, or ansi art.

Also supports SVG, HTML, LaTeX, and Markdown.

https://github.com/floooh/docker-c64

Microsoft has jokes.

I use AI to generate the pictures when natural supply is scarce. Cache them locally, address by host key hash. Use the hash as seed for bonus points.

    $ mpv -vo=sixel ~/Downloads/pirated-hollywood-blockbuster.mkv

   convert image.png six:-

Demo (running on cygwin's mintty): https://twitter.com/uguu_org/status/1631207051602042880

Probably work better if it was a comparison table, with ticks and crosses all in one column.

And the kitty protocol also supported by konsole has the exact same issue last time I tried, doesn't zoom in.

It's a simple feature, considering mintty was built on top of putty

> It's 2021, and we should be able to do litterate programming in the console, with full graphical support.

Yeah. We are stuck cosplaying computers from the sixties.

What's even funnier, even if you find a modern terminal emulator that supports features like ligatures, graphics, emoji etc. you still will be blocked by tmux. Sure - not everyone needs tmux. If you never work on remote machines, you can live without it.

But I work on remote machines all the time. I also use Kakoune text editor that defers window management to external tools (WM or tmux, but to be honest, tmux is much better). Zellij is more of r/unixporn bait than usable tool for now. So I'm stuck with text only interface.

[1]: https://github.com/csdvrx/sixel-tmux/blob/main/RANTS.md

Yes, for nine years now; bound to “Ctrl-a |” by default: <https://www.gnu.org/software/screen/manual/screen.html#Split>

You just have to have an emoji font installed (Twemoji, of course). And I use Alacritty, a Rust-based, GPU-accelerated terminal editor. How's that for "modern"? :sunglasses:

https://www.enlightenment.org/about-terminology.md

I mean, you could avoid knowing anything about the terminal state machine if you dumped the entirety of the session’s bytestream from the very beginning on each attach event, but unless you are willing to do that it seems to me that you’re going to have to track enough state to basically amount to maintaining a screen buffer (and then you do have to care about sixels).

For example, suppose the user fills the screen (maybe even with an image), then spends a lot of bytes overwriting the last line over and over (think progress bar). Either you’re tracking the first byte to have affected the current screen (and need to store the entirety of the “a lot” and dump it onto a newly attached client) or you’re trying to discard all the stale updates while keeping the last one and the initial screen contents (and that’s just a screen buffer with extra steps). (Incidentally, I suspect Muratori’s Refterm fails this test to the point of requiring a redesign, though I haven’t checked.)

I see, yeah, that’s neat. I can still imagine situations where it’ll fail, but they’re reasonably limited. (If you detach and attach in the middle of a large sixel image being transmitted, you’ll have the tail of it dumped on you in text form; if you detach from a session with the terminal set to red text, it’ll be back to default when you return; and so on.)

I use tmux exactly to split panels in any way I want. My terminal (iTerm2) doesn't give me the full power to manipulate windows/panels using shortcuts.

But unfortunately I'm gonna have to stop using tmux because it's not compatible with sixel. I spent a lot of time ricing it tho...

(I used to use tmux, but ultimately found it more trouble than it was worth.)

Which tmux pane management features do you miss in iTerm2?

So after all these years I still managed to not need tmux. Am I missing out? Everyone seems to be using it.

But it doesn’t have amazing uses that immediately make you say “wow”. The biggest I’ve found is setting up something on a remote server and being able to disconnect it. For example it can be much nicer for running a Minecraft server than daemonizing.

It's handy to know sometimes, but if I have a sane window manager at hand then I don't use tmux. They're not common, sane window managers, so I use it occasionally.

StumpWM similar concept but more powerful.

Of course you can:

1. create tmux session, create splits/panes 2. detach 3. asciinema rec 4. attach (inside asciinema rec)

This way asciinema records your whole terminal, and tmux running inside of it.

In other words, if you have it like this:

terminal -> tmux -> pane -> asciinema rec

asciinema can't see outside of the pane its running in. However, if you have it like this:

terminal -> asciinema rec -> tmux attach

asciinema sees all of it because its parent is your terminal, not tmux pane.

Similar, I keep all my work alive in remote sessions in tmux. Vim IDE, mutt, ... I would not want to re start this everyday. The best part is, I can reach my setup from office and from home and mobile. And if necessary even from my phone.

* Text selection using variuos shortcuts (usually I use it only for URL):

https://github.com/tmux-plugins/tmux-copycat

* FZF autocompletion from output, e.g. in case I want to diff some file I see changed in `git status`:

https://github.com/laktak/extrakto

[1] https://github.com/cipharius/kakoune-arcan [2] https://github.com/letoram/cat9

I feel like zellij is the future of terminal multiplexers, and I like using it in my personal machines, but I stopped using it for work after a while.

* Can't move tabs (fixed position)

* sixel support with iterm2 is meh. Yes, it shows image, but it does not look good.

* Support for undercurl seems to be missing.

* I need to sacrifice some other functionality I have in tmux.

Yes, I get some extras but it is not ready yet for me personally.

loose coupling via byte streams is great, and so is text, but we can do better than character cell terminals

if you sacrifice textuality, xpra does reattachment, multiwindow, and lowish bandwidth pretty well

Your description of sixels using words like 'cosplaying' or 'late 90s japanese cellphone' is not very technical. It seems biased. In any case, it would be better than being stuck with the 60s "text only" VT100 like protocols.

We can talk about technicalities, as some people do not like how sixels work under the hood, but... they work fine: I can play video in my terminals. I use gnuplot everyday.

Could it be done better by a protocol that would use compression and other features? Maybe. But does it have to be? Why should we waste time reinventing the wheel when locally there's plenty of bandwidth, and remotely too often enough?

Personally, I like sixels because prefer a standard that is well defined, and can't change on someone whim: I like stable interfaces.

I think users are unfairly held back due to biases like the one you demonstrated, and that seem very common in the free software world, which was the point of my rant: there is no logical reason sixels couldn't be offered today in gnome terminal if the key people making sure that doesn't happen changed their mind.

emoji support, not sixel support, is "cosplaying as a 90s japanese cellphone"

the vt100 is from 01978

my 'cosplaying' comment was a specific response to teddydd's 'cosplaying computers from the sixties'

---

i've been playing video and plotting functions from remote machines on terminals since 01993, but using x11, not sixel

i think h.264 (cosplaying as an mp4 player from 02005) is probably a better approach for graphical applications; that's what xpra uses

With these clarifications, I now understand your reply better, and it is indeed accurate: emojis are too limited.

However, their frequent use in text interfaces demonstrate the need to transmit more information density that regular text can carry.

Personally, in terminals I suggest the use of sixels, since they work well enough.

Bandwidth is never an issue for local terminals: playing videos with mpv demonstrates that.

if you send all that video data through the ddr4 bus twice (once writing, once reading) you're using about 20–25 percent of the machine's memory bandwidth, but the linux tty subsystem actually makes several copies and runs many instructions per byte [citation needed] so you can't get anywhere close to that in practice, and certainly not if you're spending a lot of time repacking bits into six-bit bytes in order to feign compatibility with the vt340, a misguided engineering dead end from 37 years ago that only supported 16 colors and 800×480 anyway

other protocols like x11 and xpra work considerably better, but x11 video reproduction on a local terminal normally uses xshm or its moral equivalent with xvshmputimage or gl. if you're on a local terminal, you might as well use x11; the only benefit to putting the pixels in the same bytestream as text is if the your terminal is connected to your drawing application over a bytestream such as an ssh connection, a tcp connection, an spi connection, or a uart

i think we could do better by designing a bytestream protocol that can minimizes copies and pixel format conversions and is therefore within epsilon of the speed of xshm, but sixel isn't it, and neither is x11 without xshm. a bidirectional bytestream protocol could include flow control to also avoid transmitting pixels that won't be visible and increasing latency due to bufferbloat

as for emoji, they don't improve information density; there are only 2666 emoji in unicode 10.0, so at a maximum they convey only 13 bits of information, and they normally occupy roughly the space of two letters like 'n', so you have slightly fewer bits per pixel. people use emoji because they are cute and colorful, not because they are ithkuil. they're not

sixel is, like the rest of retrocomputing, like code golf, and like malbolge, a technology that makes easy things hard, not hard things easy

that's a fun way to spend my time when i choose to but it's not how i want my primary user interface to my computer to work

Sixel is defined for two purposes: the first is to allow display of bitmaps. the second is to allow the definition of characters glyphs (which xterm does NOT have -- due to graphics limitations). If the glyph definition could be done, your bandwidth requirement would be reduced. Back in the 80's we generated custom fonts for PostScript (the Apple laser printer PostScript would cache the glyphs). As needed, the printer would request glyps and these would be supplied by an external driver or box (we called this the "Robin Box" for fairly obscure reasons). This system provided pre-press proofing for many customers (which would include publishers like Ziff-Davis). Please note that the speed of the communication was 9600baud (1000 cps). The image of that printer was 300x300 dpi, with some printers doing 600x600. Sixel would have been good... we used ASCII.

I like sixel -- I use xterm which gives me the option. For everything? No. but if I need a graphic, it is easy enough to use. The alternative for most would be to simply generate PostScript, and run GhostScript to view the results (typically on another system). I have NEVER contemplated watching a video with sixel... I could do it, but the player would be a "labour of love" - I would use a common decoder to produce an uncompressed bitmap (scaling and colour reduction) then convert that into frames of sixel... Just to show StarWars on a 340. But, no urge.

The main issue is that sixel offers the feature. No real reason to NOT have it... and it exists.

Try pbmtoln03, ppmtosixel, imagemagick convert and lsix

sixel is not a protocol: just an encoding. Your idea conflates the two things.

The purpose of the Robin Box? The publishers typically had hundreds or thousands of typefaces. The PostScript Printer? 15 to 50. Since RAM in the printer was limited, this approach allowed the scanning of the target typeface, conversion to outline form and production of dynamic programs executing in the printer. That are discarded but results cached. (and note that technology is within a period that, in this case, was bounded by RAM space, and typographic conventions -- all of which changed fairly rapidly).

That approach allowed hundreds of typefaces to be used on a single page! With "standard" PostScript; on a printer with only a megabyte or so of RAM available. Sixel is simply a similar tool. You really can't predict how something like would be used if it were generally available.

i just think it's a worse way to display graphics than things like x11, which is itself no shining gem

i mean worse in the sense that it requires more effort to achieve an equivalent result; it provides a worse result with the same effort; and there are some results it simply can't achieve

if you're looking for a puzzle to solve for fun, of course, those are advantages, not disadvantages; but i repeat myself

I read through that entire Bugzilla page, and IMHO the usability and technical concerns are pretty valid. You can say "well, those are not an issue for me" or "I can live with it" and that's entirely fair, but I can understand how a maintainer would want it to work really well before merging.

Also, I believe it did get merged in 2020? The current status is a bit unclear to me.

Same. Sixel support in gnuplot is very useful: it’s great to have decent graphics without relying on X with dodgy vpns. It’s probably not the most useful feature, but it was a significant QoL improvement when I found out the Sixel terminal.

* show multiple terminals on screen. The best solution is to use a tiling window manager. Both MacOS and Windows have limited windows tiling capabilities. * run long term TUI applications (like editors). The best solution is mosh. * run long term CLI applications (like shells). The best solution is dtach.

tmux helps all 3, but not particular good at either.

iTerm2 on macOS has some nice tmux integration[1]. Basically, you run a tmux session (using tmux -CC), but the actual window management on the client side is handled by iTerm2. This works pretty nicely with the tiling WM (Amethyst[2]) I use on macOS.

If anybody is aware of Wayland compositors that integrate similarly, please let me know. I'd love to be able to do the same on my linux machines.

[1]: https://iterm2.com/documentation-tmux-integration.html

[2]: https://github.com/ianyh/Amethyst

I'd rather not live without it even on local stuff. It's way too good. Yes, I already use a tiling wm, and no it's not the same.

This is a more promising and updated sixel branch of tmux, which is slowly getting updates propagated to mainline.

Do not assume my terminal!

Next you'll want accessibility, alt text for blind readers, DPI awareness, responsive scaling on different screens, etc.

Convenient terminal tools already exist for viewing formatted text like Markdown or HTML, or viewing binary data like ELF files, or showing live dashboards like htop... but images still require launching a separate application. And this is all data that you might produce while working solely within the terminal--only to have to visit GUI land to see what's in a PNG file. It's a pretty obvious gap.

That said it's disappointing that the best we can do is Sixel, a hilariously inefficient and ancient format.

It has been in XTerm since the 20th century, and terminal emulators for MS/PC/DR-DOS like Reflection gained Sixel support somewhere around 1989.

Don't be fooled into thinking that "the terminal" is what one can do on the kernel virtual terminals of Linux, FreeBSD, or SCO Xenix. The reality is that the massive reinvention wave of the 1990s and 2000s has actually lost you functionality that was in real, hardware, terminals and contemporary terminal emulation programs of the 1980s, and given those who never lived it a very blinkered idea of terminal functionality based upon 1960s TeleType terminals that was a little outdated even when Unix itself was invented.

It's there: https://www.arewesixelyet.com/#mlterm

Green: moral

Red: immoral

Red green colourblind: amoral

For instance, in Mac OS, under Accessibility there are 'Color Filters' with a dropdown for presets for a few common types of colorblindness including red/green. It is a filter that is applied systemwide at the graphic output level.

I think this is the proper approach (device-level color filtering) versus each app attempting to serve all, as that could lead to double-filtering.

https://sw.kovidgoyal.net/kitty/graphics-protocol/

Look at the “The transmission medium” section. I’m reasonably familiar with what can go wrong when messing with files you shouldn’t mess with on Linux, I’ve worked on the various security mechanisms that can help, and I would not want to implement this. If someone wanted me to consult on implementing this, my advice would be, first and foremost, not to.

There isn’t even a “security considerations” section in the document!

There's an escape code to enter "sixel" mode, then base64-style data representing 1x6 pixel bitmaps, then an escape code to get back out.

No rasters that might overdraw unexpectedly (security considerations!), no mechanism for out-of-band data reading (security considerations!), but also none of the extra magic that something like kitty would provide.

I could live happily with Sixel being universally supported in terminal emulators.

Me too!

If I want fast graphics from a local application, there are a couple of widely available ways to make it work. X11 and Wayland come to mind :)

My terminal does not live in a mountain fastness, and it’s not as exposed as a web browser, but it should at least try to make it safe to feed it untrustworthy input.

I'd like to play with terminal graphics in a an app I'm working on. Does anybody know if a terminal which supports:

- wayland - sixel - copying substrings without a mouse

I know I can just pipe to wl-copy, but sometimes the right chain of sed & awk is less convenient than the right chain of vim highlight movements.

https://iterm2.com/documentation-images.html

This is not to say that I am against sixel. As someone who was doing graphics on terminals in the 1980s, sometimes by accident when I catted the wrong file, the idea that "terminals don't do graphics" seems blinkered and ahistoric to me. They do and did, in my direct experience. I'm simply not targetting that as a feature in a program that has the specific goal of matching specific kernel built-in emulators. You know where XTerm is if you want it. (-:

On the gripping hand, my terminal emulator does Unicode (as do several of the kernel built-in ones, albeit with severe limitations), and so all of the things that one can do with Unicode pseudo-graphics are possible on it, from MouseText windows through progress bars using the 1/8th block glyph set to PC-style line drawing.

https://news.ycombinator.com/item?id=35940691

Echoing another comment here, it doesn't work in tmux though which is pretty heartbreaking. But viewing beautiful maps in the terminal is amazing, and works perfectly.

(I'm using the alacritty branch that supports it, worked perfectly.)

  echo $'\EP0;0;0q"1;1;6;6#0;2;100;0;100#1;2;0;0;100#2;2;0;100;100#3;2;0;100;0#4;2;100;100;0#5;2;100;0;0#6;2;100;100;100#7;2;0;0;0#0!6_$#1!6O$#2!6G$#3!6C$#4!6A$#5!6@-\E\\'