News for hackers. Stuff that matters.
Re:Text Processing in the Shell by JdeBP on Tuesday March 17, 2020 @07:53AM (#22603975)
... and pretty much all of those were addressed, outwith Unix, by the evolution of the 1960s terminal I/O model into the console I/O model during the 1980s.

You even forgot to mention one of the things that was addressed: input. Terminal I/O input, done properly, requires a full ECMA-48 decoder state machine, with bodges to accommodate non-conformant warts from the Linux KVT, SCO Console, and RXVT. This is all too often not done properly, because people do not realize that there is ECMA-48 in both directions; and is a mess, just looking at function keys alone and not even accounting for keypad application/normal modes and a mouse/locator. Console I/O evolved into uniform input event records for HIDs that did not require state machines to decode.

Note that the lack of a layout system is only applicable to character-mode terminals. Block-mode terminals are a quite different kettle of fish.

Re:Text Processing in the Shell by hnlmorg on Tuesday March 17, 2020 @08:27AM (#22604197)
> ... and pretty much all of those were addressed, [outside?] Unix, by the evolution of the 1960s terminal I/O model into the console I/O model during the 1980s.

> Note that the lack of a layout system is only applicable to character-mode terminals. Block-mode terminals are a quite different kettle of fish.

Indeed but the point isn't "are these solvable problems?" but rather "why are we still using archaic tech?"

Designing a solution to those problems is actually the easy part. It is shifting the ecosystem away from TTYs that's hard.

> You even forgot to mention

It wasn't intended as an exhaustive list :) There's plenty more issues I hadn't raised.

---

In an ideal world I'd love to see UNIX terminals reinvented. The reality is things are "good enough" for most people that they simply don't notice most of the issues and migrating to the next evolution of UNIX terminals would mean a break in backwards compatibility which will be more disruptive (initially in a negative way) than making do with the warts we currently have.

Re:Text Processing in the Shell by dublin on Tuesday March 17, 2020 @12:12PM (#22606377)
Check out Microsoft's new Windows terminal (https://www.hanselman.com/blog/ItsTimeForYouToInstallWindows... ) It may be the most modern, capable, and yet compatible terminal I've come across, and works well regardless of the environment you want to run it in/with. (To be fair, a modern Windows terminal is a good 20 years overdue, but MS deserves credit for finally getting it right...)
Re:Text Processing in the Shell by JdeBP on Tuesday March 17, 2020 @04:48PM (#22610236)
It doesn't really support all of the basics as exemplified in https://github.com/jdebp/terminal-tests/blob/master/PowerShe... and https://github.com/jdebp/terminal-tests/blob/master/PowerShe... yet.

https://github.com/jdebp/terminal-tests/blob/master/PowerShe... is arguable, and depends from whether one sides with old actual DEC VTs or the ECMA-48:1986 standard, which the DEC VTs didn't keep up with. There are at least two existing terminal emulators that side with the new 1986 semantics, nowadays.

It really isn't anywhere near the most capable terminal emulator right now. This is acknowledged by its developers, and they even have long to-do lists of missing stuff, both as compared to a DEC VT and compared to the likes of XTerm.

Re:Text Processing in the Shell by hnlmorg on Wednesday March 18, 2020 @04:48AM (#22614738)
If it's backwards compatible with TTYs then it probably hasn't solved any of the problems I mentioned and there's already a plethora of nice terminal emulators out there (which is why I say most of the "ugliness" is hidden from everyday users).

It should also be noted that Windows makes a few mistakes when it comes to terminal design too:

- For starters cmd.exe is both a shell and terminal emulator and it's impossible to separate the two.

- To compound things, many common commands like dir, rm, etc are shell builtins (this will be a throwback to DOS). So you cannot even use an alternative shell on Windows without having to either invoke cmd.exe or rewrite existing utilities.

- And if that wasn't bad enough, cmd.exe builtins do not read from STDIN. They instead use DOS syscalls to read keyboard input.

- As you've probably guessed, cmd.exe isn't the only culprit which does this. Any "Windows" command line program designed for or which uses DOS APIs will not follow the standard streams idiom. Any command line software written for NT, however will. So you end up having to write all sorts of really nasty hacks just to get the command line working on Windows (far far nastier than any of the hacks that happen on UNIX/Linux).

- Then you have Powershell, which is an entirely separate command line in its own right and largely - though not completely - incompatible with cmd.exe.

- And WSL, which is also incompatible with cmd.exe and Powershell.

At least on UNIX/Linux, you have one terminal methodology. From there you can use whichever terminal emulator you want, whichever shell you want, whichever programming language to write CLI tools and/or whichever CLI tools you want to download. Where as on Windows you have 4 competing standards which don't cooperate well.

Re:Text Processing in the Shell by JdeBP on Wednesday March 18, 2020 @09:22AM (#22616321)
It behooves one not to make egregious mistakes when talking about the mistakes supposedly made by Windows. cmd.exe is not a terminal emulator at all, and does not make DOS system calls (it being a Win32 program) at all.

* http://jdebp.uk./FGA/a-command-interpreter-is-not-a-console....

And the fact that CMD and PowerShell provide different interpreted languages is no different to the Korn shell, tclsh, and Perl providing different languages.

Re:Text Processing in the Shell by hnlmorg on Wednesday March 18, 2020 @11:48AM (#22617940)
> It behooves one not to make egregious mistakes when talking about the mistakes

Admittedly it's been a little while since I last played around with custom shells and terminals on Windows and I had also rushed my post so you're right that some details were wrong but you're just as far off with your corrections as the points you were criticising so you're really not in a position to be making platitudes about egregious mistakes.

I didn't say cmd.exe was a terminal emulator, I said it was multiple layers including the terminal emulator but not exclusively the terminal. Ok, techically it's conhost.exe that provide the terminal emulation, I'd lazilly lumped that together with cmd.exe because cmd.exe depends on conhost.exe when not run headless. The former requires the latter so you can't just drop cmd.exe into another terminal emulator and run it (other people have tried and there's extensive blog posts about the hacks they've had to do to get it to work, like running conhost.exe off screen).

If you want to be 100% technically accurate then cmd.exe is actually not like any of those things we've described. It's certainly not equivalent to Korn or other language REPLs as you stated. In fact the "language" part of cmd.exe is barely a macro language (again, due to it's DOS heritage). Plus shells orchestrate with byte streams where as NT's streams work very differently and cmd.exe doesn't even behave correctly when used as a CLI tool (which I'll get into below).

You're right that cmd.exe itself doesn't make DOS syscalls, as said aboveI was rushing my post which caused me to conflating two points. What I really meant to say was:

1. cmd.exe builtins read from the NT console API's stdin. Which means you cannot fork out to cmd.exe as a CLI command because any prompts ("Are you sure you wish to delete" type things) just whiz straight past without pausing for input. This was highly annoying when I was developing my alternative Windows shell and wanted to make use of rm, copy, etc rather than having to write those commands all over again.

2. Windows, and cmd.exe by extension, supports running other console applications which don't use NT's console streams because they favour some of the other hacks used in the DOS days. This means those applications also don't work with alternative shells let alone alternative terminal emulators.

Also I think it's disingenuous citing your own blog post as a source. I could link you to the Github repository where I've had to put in numerous workarounds for the shell I've written to work with Windows. But instead I'll link to something a little more recognised:

https://devblogs.microsoft.com/commandline/windows-command-l...

(I did have a hunt around for the blog posts from other developers building console solutions for Windows and the similar problems they've ran into but since it was around 5 years ago when I gave up first party Windows support, those blogs are now lost in the mists of the ether).

Re:Text Processing in the Shell by JdeBP on Wednesday March 18, 2020 @09:31AM (#22616409)
You've got it backwards. This is the reality where UNIX (et al.) terminals were reinvented, throughout the 1980s as I said. This is the real problem, people going around talking of making a "better terminal", and when pressed talking about things that the world actually did make better, decades ago, and treating as speculative I/O paradigms that demonstrably exist today.
Re:Text Processing in the Shell by hnlmorg on Wednesday March 18, 2020 @01:23PM (#22619008)
I know full well terminals have been "reinvented" but, and as I've already said to you in that very post you're replying to, solving these problems isn't the hard part. It's getting people to migrate to it. The simple fact is UNIX nor Linux doesn't adopt any of those designs you've been discussing and this conversation is specifically about the POSIX command lines. So you can argue all you like about how x does y better but it doesn't mean shit if nobody can actually use x.

I've been doing this for 30+ years. I've written my own terminal emulators and UNIX shells. You're not the only nerd on here so take a step back and listen to the points people are making before assuming they need to be re-educated :)

Re:Text Processing in the Shell by msla on Tuesday March 17, 2020 @11:55AM (#22606145)
> the evolution of the 1960s terminal I/O model into the console I/O model during the 1980s.

Are you talking about MS-DOS-style memory-diddling to achieve things like colors and reverse video?

Re:Text Processing in the Shell by JdeBP on Tuesday March 17, 2020 @04:31PM (#22610008)
Re:Text Processing in the Shell by msla on Tuesday March 17, 2020 @06:47PM (#22611413)
So, to quote:

> Platforms with "consoles". These platforms provide a concept of a "console" to applications programs. Consoles support direct screen addressing, to the level of character cells at least, and are accessed through an API that is a first-class part of the overall system API.

> Platforms with "terminals". These platforms provide a concept of a "terminal" to applications programs. Terminals are not directly addressed, but are communicated with via byte stream communications protocols, involving control characters and control character sequences.

In short, you're deliberately being non-specific. I'd claim that a terminal system plus ncurses is a flexible and reasonably efficient console which is portable between many different systems, on the hosting end and on the client end. I could claim that the IBM block-mode "terminals" are consoles if the system is taken as a whole, but such things are markedly less flexible than what you can accomplish with ncurses, albeit more machine-efficient on the host side.

As far as first-class APIs go, you can argue with others about the kernel-vs-OS distinction, and the irrelevance of unbundling in the Open Source world. In short, shipping with ncurses is no more "odd" than shipping with Gtk or, say, a web browser.

Re:Text Processing in the Shell by JdeBP on Wednesday March 18, 2020 @09:42AM (#22616519)
Actually, I'm the one using the specific terminology (from Win32) here. A console is a specific I/O abstraction, widely known as such. Whereas "I'd claim that a terminal system plus ncurses is a console" is your own idiosyncratic nonce definition, in contrast.

Console I/O has a demonstrable evolution over the course of the 1980s, as I have already explained several times, a lot of which was to address the shortcomings of the 1960s terminal I/O model.