Bwahaha... is this your first encounter with backwards compatibility?
Changing this would be very hard, but this has nothing to do with somebody embracing some grand philosophy, but with tons of existing software relying on Unix like systems to, well ... be Unix-like, because that's what the software was written for.
Invert the shell and terminal: every shell command (with unredirected streams) gets it's own pty.
- Like the those repl "notebooks" you can scroll each output separately, without hacks.
- Backgrounded processes won't spew garbage when you are trying to type. They just continue running in their terminal. You can even background something like vim or a game that takes the whole terminal, and it's just fine!
- Rather than hacking scroll back onto processes that might use tty stuff in expected ways, just straight-jacket them by giving them plain pipes instead! The OP exploits are then opt-in impossible.
- Having powershell-esque "post-text" commands that dispense with the bullshit are far easier to manage: also give them regular file descriptors and negotiate protocol or whatever, just like the above, rather than a pty.
- stdin can be a nice form with submit button rather than fixed (usually line) buffering policy.
- shell might even be simpler as there is less careful juggling of shared resources except where the programmer asks for it.
I also use ssh multiplexing instead of tmux for extra shells. When I do need to use tmux, I hate that read only mode prevents scrollback. The above builds nicely upon this:
- The persistent remote shell has as many ptys as needed before for chained commands which might still be running, or separate "sessions"
- The local client can use native UI elements for everything, no TUI jank.
- Scrolling over many (live or saved) terminals, the normal scrollback case, is possible and very side-effect free.
Anyone see any issues? I don't for anything I do daily (git, coretuils, etc.)! I don't think I have time to make this, so please someone else do.
Focusing on this point, you can background vim just fine right now - I have tons of vim processes backgrounded all the time.
Perhaps for most use cases involving vim it doesn't make that much difference—what would it be doing in the background anyway?—but there are other TUI programs that could be doing useful processing in the background if they didn't need to share the pty with other processes.
The problem, of course, is this either breaks existing software or requires new software to be written from scratch to utilize the new frameworks. This was (one of) the reasons plan9 never gained much traction, and the big hurdle for any contender to the TTY/shell throne.
(It's also worth pointing out microsoft themselves reworked their terminal interfaces to be more unix-like for this very reason -- software compatibility)
Mentioned this comment here: https://blog.williammanley.net/2021/03/31/seen-on-hn-invert-...
You’d need to implement a new protocol between terminal and shell. Maybe the shell would be in charge - allocating PTYs and multiplexing the output back to the terminal. Or maybe the terminal would be in charge where it would allocate PTYs for children and send them to the shell using FD passing. There are advantages both ways.
Integral to its success though would be getting the support for this new protocol upstream in bash. Bash is the most widely deployed shell and the only way to get to the point where you can log in to a new machine (or over SSH) and expect this to “just work”. Without that it would remain a niche and could join the graveyard of other improved shells/terminals that approximately no-one uses.
As such the protocol would have to be minimal and easily implementable in C. Systemd’s protocols could be an inspiration. I’ve implemented both sides of sd_notify and LISTEN_FDS before, in more than one programming language and it was very straightforward.
As long as this interface remains undisputed, "everything is text" is a feature, not a bug. Arguably, one of Unix's greatest.
Terminal sequences, unicode all violate the identifier rules. There is no u8ident library to sanitize this input or output.
Unicode standardises identifier rules. The default identifier as per UAX31-R1 can be checked with a regex:
perl -E'say "my_identifier" =~ /\A \p{XID_Start} \p{XID_Continue}* \z/x'
Btw your snippet does not work for identifier validation. There is much more needed to pass the identifier security guidelines.
But here we just need a simple terminal escape sequence stripping library. Unicode is a bit harder. Only Java, rust and me did that.
Look at it the other way round. Embrace text. Text is not the problem, non-text is. You'll take text from our cold, dead hands!
It's not that uncommon for ol' C hackers to directly write those stack and heap locations out to disk and call it a "file format." Trouble is, you're almost entirely at the whims of your platform and compiler as to what the actual layout of that is.
If you're thinking, "well that's dumb, why doesn't C have a standardized representation for those in-memory objects that hides platform differences", it does: printf and scanf.
Text isn't necessarily a great answer to that problem, but it definitely is an answer. Others include packed structs with htonl and friends and low-overhead serialization formats like protobuf, Thrift, and Avro. Inside, say, Google, you have "everything is a protobuf" instead of "everything is text," and it does end up working roughly as well as you might expect. That is to say, reasonably well, but with its own sets of problems that people won't ever stop complaining about.
If you are parsing "ls -l" output, you are doing something wrong.
Use your language built-in features, every language has them (for example in bash, use *-expansion and [-commands). If they don't work for some reason, there is "stat -c" and "find .. -printf", which both produce text which is absolutely trivial to parse.
Of course, ls does more than just list file names so it can be tempting to utilise its features. coreutils ls has (relatively) recently received an additional output format that can be unambiguously parsed, but that's as far as the portability goes.
> Use your language built-in features
The issue is that current shells _don't_ have a built-in structured data type for e.g. a `struct stat`. But why not?
`ls` calls some APIs and gets some in-memory `struct stat`s populated by the kernel. Then it throws away 90% of the data that the kernel copied to user space and then serializes it as text. Why not pass the structs themselves to the next process? We can't currently (except with powershell?), so you have to write actual code.
This is a "bright line" between code and shells that could very well be blurred, but it would take an agreed-upon serialization format for posix-y data structures.
This is why in shell, if you want programmatic "stat" output, you use "stat" tool, not "ls". "ls" prints all at once. "stat" has a custom output format, which means you print exact fields you want, so your intermediate results are concise and readable. And since every element of the "struct stat" is an integer, a simple space-separated format works very well.
(that said, I would not mind seeing more tools print JSON or JSONlines. I add this functionality to many of the tools I write, and it is pretty powerful in conjunction with jq)
Because then the output of ls can be used by programs that don't know what a list is, or what a file is; and this is breathtakingly beautiful.
Structural data can still be serialized easily, take json for example which’s spec fit on a napkin.
`ls` is for you, the human.
system("ls");The moment you want to do something else with the text, you have to parse it in some way. At that point it's no longer text, it's a poorly specified serialization format.
It's not really good, but calling it abysmal is just too negative in my opinion :) It does get a lot of things right after all, especially in the "not everything is text" department. Also a lot of things wrong (my main gripes would be a lot of the syntax in general, case insensitive, usually more than a couple of ways to achieve the same thing). So in the end the learning experience/curve for me was the same as for bash (or languages like C++) and just as long/steep probably: feels like suffering until you've learned all the caveats by heart and in the process of doing that also learned how to actually use it, then it becomes ok to use. After that process I'm leaning towards favoring PS over bash though. Now it's possible there are already other shells which get more things right but I really doubt that in my lifetime I'm still going to spend (waste?) time on learning yet another shell. Or it must be convincingly good and provably fast to learn :P
Also the commands were super long. Sure there was the ability to alias them to something shorter but still....
That being said... powershell has a lot of potential.
(Also I’m woefully out of date... I haven’t touched powershell in a decade. I hope somebody tells me it’s all better now)
Windows Terminal is gr8 and removes all those problems. You have alternatives too like ConEmu.
Long commands are OK, use aliases. Bash has long command arguments too and nobody complains.
Saying Powershell has a lot of potential is kinda lame - its best of them all.
I do not expect text to be significantly replaced any time in the next 5 years. Accessibility - screenreaders, increased text size with text reflow - is one massive reason.
Adding a --json flag is just the start. Ideally, other grep-like programs would need would want to emit the same format. But what happens when some grep-like programs have additional features?
Suggesting a structured representation as an output format isn't a panacea. It's just the start and it's not clear at all to me that it would be better than what we have now.
- the pattern that was searched for
- the search options from the invocation (case, line folding, etc)
- the text of the matched segment
- the byte range of the matched segment within the file
- the line number(s) of the matched segment within the file
- the file name
- contents of any (named) capture groups from the search patternI also worry about a bad standard becoming popular a la most of the web
I'm also not extremely familiar with the concept, so at the moment, I'm reading this document to understand it better https://www.varonis.com/blog/how-to-use-powershell-objects-a...
Isn't the real solution to do away with the idea that we need to remain compatible with VT100 video terminals?
We've learned a lot since the unix philosophy was created. One of those things is that, if you want software that lasts, software that performs well, you need to be strongly typed.
Plain text is as far from strongly typed as you can get. Passing plain text between programs in 2020 goes against everything we've learned in the last 50 years.
In theoretical terms, strong typing every, always is preferred. In practical terms, this devolves into a Java-esque nightmare of what's-the-type-for-the-parent-type-of-the-thing's-type absurdity. Where people stop using your system because they don't want a second job learning its Borges-esque comprehensive type encyclopedia.
In practical terms, bare minimum typing that prevents 80% of errors, but doesn't require gyrations to cover the remaining 20% works. And given a choice between the two, more software is going to be built in systems that function this way.
Does it break all the damn time? Yes. Does it still get more done? Yes.
I would have to say the opposite is true.
It's text based interfaces that have stood the test of time, and now even flourish in HTTP. We have more widely used text based interfaces than ever before where REST,HTML, XML and JSON seem to rule most machine-machine communication.
Strongly typed binary RPC such as DCOM and CORBA however are now nearly gone.
Unix itself, for example, won't last even 10 years. Look for it to be gone by the mid-late 1970s as it gets replaced by strongly-typed objects passed directly between applications to control our fusion-powered flying cars.
What will computing look like in 500 years? Do we really believe that we reached the pinnacle of OS development in the 1970s?
I know but my point is that the interface itself is still text.
Terminal emulators are, far and away, the most archaic computer technology we still use daily.
Believe me when I say, it's a very hard problem to solve since everyone has subscribed to Unix's philosophy of "everything is text".
As much as I dislike Microsoft's philosophies on software, they had the right idea with Powershell. Its execution is just abysmal.
Expect this stuff to change drastically in the next 5 years.