Absolutely agreed, shell is in many respects terrible.
>I don't think that a shell should be a complete programming language.
On the contrary, I think shell should be a more complete programming language! Drop the stringly typing and add actual types (hence eliminating 80% of bothersome awksedgrep magic; yes, no need to tell me it's a real tall order), add proper error handling...
I truly value simplicity where possible, but I think that the primary interface that I use to communicate with my computer should be as powerful as possible. Numerous times I've built up a shell pipeline only to realize near the end that I need to do something that shell is horrible at, and had to redo the whole thing in Python from scratch. I cannot really see a reason to limit it.
>Some of the most commonly used utils like awk '{ print $2}', sed, grep, sort, ... should be included out of the box to improve performance and to provide consistency across distributions.
UNIX "philosophy" got a lot of things wrong, but not this one. These common utilities you mention are good especially when they're separate, because it's not the shell's job to improve performance, provide consistency, etc... of a dozen and a half different utilities, a number sure only to grow in time as people discover what commonly used thing they want in their shell. Though I'll give you that `awk '{ print $2 }'` should definitely be its own utility or a built-in.
[1] https://code.joejag.com/2020/a-month-with-powershell.html
For example, find all the .py files under the current directory, find the ones that have changed in the last day, and then print the lines (with their filenames) that define classes:
ls -fr \
| select (f: f.suffix == '.py' and now() - f.mtime < days(1)) \
| read -l \
| select (path, line: line.startswith('class '))
(Yes, this is doable with find and awk, but this is just an accessible example that gets across the idea of Python functions on the command line.)
You don't even need awk for that one:
grep ^class `find . -mtime -1 | grep py$`
read -c foo.csv | sort (*x: int(x[3]) + int(x[6]))
Partly, I was also scratching an itch: I prefer using Python to exploring the sublanguages of various shell commands. Much cleaner, in my opinion.
Yes, that's a much better example. Here the "obvious" shell solution involves using awk to compute the sum of the two columns and putting it as the first field, sorting by the first field, and then removing that field. I guess a "rosetta stone" of such examples (e.g. on the frontpage of the site) would make a strong case for the interest of marcel.
awk -F, -v OFS=, {print $2+$3, $1, $2, $3}' foo.csv | sort -n | cut -d, -f2- awk -F, -v OFS=, '{print $4+$7, $0}' foo.csv | sort -n -k1 -t, | cut -f2- -d,
awk -F, -v OFS=, sets the input and output column separator to comma, '{print $4+$7, $0}' outputs the sum of column 4 and 7 before the rest of the line.
sort -n -k1 -t, sorts the file numerically on column 1, with comma separator.
cut -f2- -d, removes column 1, with comma separator.
This is of course not robust for general CSV files, but I don't think OPs marcel is either. A robust solution requires a proper CSV parser.
The biggest warts I see in the classic unix solution is that all the tools use different flags for the field separator.
edit: if you know that the csv doesn't contain tabs, you can omit some flags for a more concise
awk -F, -v OFS='\t' '{print $4+$7,$0}' foo.csv | sort -n -k1 | cut -f2-
I haven't tested corner cases, but marcel relies on the python csv module, which is probably better than any initial attempt at a parser that I could write in an hour.
This is what I meant about sublanguages. Many people, (myself included), would need to go to the man pages to find the necessary arguments to awk, sort, and cut. I find it much easier to just write a little Python, even if the end result involves more typing.
<foo.tsv awk '{print $4+$7,$0}' | sort -n -k1 | cut -f2- from marcel.api import *
for path, line in (ls(file=True, recursive=True) |
select(lambda f: f.suffix == '.py' and
now() - f.mtime < days(1)) |
read(label=True) |
select(lambda path, line: line.startswith('class '))):
print(f'({path}: {line})
read(label=True)It's a shell embedded in the Racket programming language. If you're familiar with Xonsh, it's similar, but both more powerful and less polished. It allows easy, recursive mixing of shell code (not posix compatible, but with a similar feel) and Racket. Its interactive mode is not polished (I need to write a better line editor), but it works, and it's great for programming.
I haven't worked on it much lately (I need to wrap up other things to finish my PhD), but this weekend I added support for user-programmable substitutions with automatic cleanup. As an example, beyond common substitutions like process substitution (IE <() and >() in bash), I added a demo “closure substitution” form that allows you to send Racket functions to `find -exec`. Importantly, this is something that a user could add, because Rash is super extensible and malleable.
Shell is a great DSL, but there are huge advantages to having an embedded DSL rather than a stand-alone DSL. Rash inherits all the cool features of Racket and can be mixed with other Racket languages (eg. Typed Racket, Honu, etc), has advanced features like first-class delimited continuations and the world's most advanced macro system (which makes Rash possible), and any shell script can import functions from Racket's catalog of third-party packages. Embedded shells like Rash allow a shell script to be copied from interactions like you do with Bash, but then grow past the “rewrite in Python” stage gradually with no rewrite -- just a gradual transition from being more shell code to being more “normal” code.
Though I'm kicking myself for continually not getting around to rewriting all the documentation, which is still very poor.
Which is also one of the reasons I prefer using Vim over Emacs; I can embed Vim in any terminal multiplexer of my choice and combine it with various terminal programs. Every single part of that environment is easy to understand and easy to replace with another program or script. It's less homogenous and consistent, but I am fine with that tradeoff. As one might guess I am not a big fan of IDEs.
If these design points turn out to be over-constrained and have to make major compromises, yes the next thing I would try would be a small separate language for shell.
So far nobody has complained about any of this, I'm guessing because it looks very familiar, and that was intentional. Oil takes some pains to literally look like shell + Python syntactically, with better semantics.
I noticed that a few other shells are having problems with this command/expression distinction, and I discussed it like 3-4 years ago with Ilya Sher (of NGS) and a few other people. They were also having the same problem.
For example, does / mean a path separator or the division operator? Does * mean a glob or a multiplication? In Oil, this is no problem. It's obvious depending on the context.
Though I'm interested in more feedback on this, and the latest release is available to try as always :) https://www.oilshell.org/release/latest/
---
The awk and make integration is still doable but not done. Awk might require a notion of "lazy expressions" or "lazy arg lists", which would be shared with dplyr-like functionality. Oil's Zulip is open for discussion on these ideas :)
[1] https://www.oilshell.org/release/0.8.5/doc/command-vs-expres...
Marcel is also somewhat verbose. It exposes Python functions on the command line, so when you need to write such a function, it starts getting long. E.g. if you want to sort files by last time modified:
ls | sort (f: f.mtime)
bytime = [sort (f: f.mtime)]
ls | bytime
Perl had very good performance, despite being a "scripting" language, because so many of the tasks that a typical shell script might need to do were supported by built in features in the language (for example handling regular expressions, interacting with command line arguments, exec'ing other programs).
Today, once a shell script requires very much logic, I'm inclined to use Python. I know Python and find programming in it easier than Perl and bash and other shells. Furthermore, other programmers understand Python better than bash.
Four Features That Justify a New Unix Shell http://www.oilshell.org/blog/2020/10/osh-features.html
Absolutely! Some work should be put into making it way more intuitive. Having to remember what every flag means (which is different in every app!) is not intuitive at all.
By default there should be some sort of intelisense auto-complete which can also provide guidance on what on earth all the flags mean, and maybe eventually it could tell you what it expects to do before you run the command. For most users otherwise it ends up being "paste in this command you googled with a load of random flags you don't understand and won't remember"
Why does terminal have to look like terminal? Like why is the 1980's ASCII telnet style the only way to do this?
The ASCII fish image on boot and fully ASCII menus only reinforces again that this isn't a modern interface, its an improved 1980's interface.
Why, for example, can autocomplete not look like this? https://code.visualstudio.com/docs/editor/intellisense
Why do progress bars when you are downloading from pip not look something like this? (still in-line with the terminal like a sparkline) https://docs.microsoft.com/en-us/windows/win32/uxguide/progr...
Like surely everything doesn't have to be SO 1980's if we want shell to stay relevant.
for real though, after looking at your examples I feel that features like those would fall on your terminal emulator to bring to the table, not the shell itself.
specifically, I don't want to imagine trying to connect to a headless server during some crisis over bad wifi and have the shell think it needs to send a bunch of graphics to me.
Bash (and other shells too) is like it is, because it's an accretion of 50 years of history rather than a singular top down design.
People joke about Emacs being an OS but it’s really just a different kind of shell, oriented towards text editing instead of issuing commands.
I am not sure if we should care about the differentiation between TECO-Emacs on one hand and Elisp-Emacs on the other.
You can have newlines in filenames, but you can also pass binary data with \0 if you want (it's 8 bit clean). Or you can use read -0 for a NUL delimiter, and eventually length-prefixed blobs with netstrings.
The way I now think of it is that Oil should have all 3 solutions to "the framing problem" in networking: delimiting, escaping, and length prefixing. So you can convert from one regime to the other by using shell.
PowerShell; although you have to use long form to do it:
PS C:\> ${var with spaces} = 1
PS C:\> Get-Variable *space*
Name Value
---- -----
var with spaces 1
PS C:\> ${var with spaces} + 2
3In SQL, notably. Such identifiers need to be quoted, of course.
ISTR that many dynamic languages allow this with dynamic assignment features, but you often need to use similar dynamic access functiona rather than simple variable access to access them.
Ruby also allows whitespace if it is unicode but not ASCII whitespace, without use of dynamic assignment/access methods.
SQL allows spaces in object names but requires quoting, similar to shell.
No, it wouldn't; you'd just get EINVAL if you pass a filename containing "\x20" to open or other syscalls. If a application wants to use "\xC2\xA0" in a filename, it can do that, or it can not do that, same as currently. Same applies if it wants to use, say, "\xAA\xFF".
My dream OS would forbid path names that don't match \w in grep.
What do you mean exactly? If you have pipes and lists (e.g., the lines on a file) you are Turing-complete and people can and will make arbitrary programs. How would you like, precisely, to restrict the shell language so that it is not Turing-complete?
I am not the author of that piece, but the analysis is wonderful.
I don't think that a shell should be a complete programming language. If you need a programming language, then better use one. There is xonsh if you are looking for something like this.
I think there should be a better bash with an very clean and consistent interface. Some of the most commonly used utils like awk '{ print $2}', sed, grep, sort, ... should be included out of the box to improve performance and to provide consistency across distributions.