This abomination reminds you to use a proper scripting language.

... or perhaps, as the article suggests, that parsing was hard when we first started doing it decades ago and not everyone got it right.

If it wasn't abundantly clear, this is not an issue with any of the shells today.

> If it wasn't abundantly clear, this is not an issue with any of the shells today.

The least-common-denominator aka /bin/sh and its syntax will stick around for decades to come. You can't rely on any of the "modern" shells to be around, particularly if you write scripts that target a multitude of unix-like systems (macOS, xBSD, Linux).

It's amusing that parsing theory was already at least 1-2 decades old by the time they made this mistake (consider that ALGOL 60 had a complicated grammar, introducing BNF too). I'd chalk it up to this being a shell language that was evolved rather than designed.

I wouldn’t say that parsing is hard (precedence parsing is especially not hard), but instead that logical operators and grouping inside test aka [ were both a bad idea (given a sufficiently complex expression, I expect you can still confuse a human as to what’s an operator and what’s a string even if POSIX is precise enough to dictate a single interpretation) and unnecessary (the shell itself has perfectly serviceable !, &&, ||, and grouping, though the ! is a bit awkward and seems to be a later addition).

If you omit those, a fairly intuitive unambiguous parse is very easy: if we’re [, check last argument is ] and throw it away; if one argument, return whether it’s empty; otherwise, if two, the first is a unary operator, evaluate and return that; otherwise, if three, the second is a binary operator, evaluate and return that; otherwise fail. That’s it.

The OP gives two examples where it is not fixed, because bash is so poorly designed that the fixes are partial hacks.

> If it wasn't abundantly clear, this is not an issue with any of the shells today.

I really doubt it. If the comparison parameters are undistinguishable from the operators, it's only a matter of enough creativity and will to find new ways to break it.

LOL. And what how would one run that proper scripting language. On top of a shell? With a userland and kernel built with shell scripts? Or maybe Microsoft Windows? Perhaps with Windows one can avoid the Almquist shell. Otherwise, it's still there whether one chooses to use it or not. Scripts using this convention are common, at least on NetBSD.

This is such a bizarre response. A Python* script, to give one example, does not—at least in any meaningful sense—run on "top of a shell". The shell will accept the command string and fork the process, but that has fuck-all to do with the design of the shell language being discussed here, and the whole thing happens to be incidental, anyway—you could completely remove the Unix shell from the equation by just making the Python interpreter your shell if you wanted, which would sidestep every element of the already-tenuous line of reasoning that you're using here to try to make an argument...

* which I abhor, by the way—just like GNU abhors man pages—but that doesn't make your argument any less absurd

he probably meant on the terminal but he's just LARPing like every other UN*X enthusiast. they like to create false dilemmas like "legacy" and "embedded"* to justify their bloated poorly implemented string flinging machine that has another 100KLOC to parse the input line (readline)

imagine someone pitching shell programming today. the response would be: LOL so you mean each language will have a common subset of syntax that supposdely works the same across all of them? how can anyone ever program that way?

* ironically, a big part of why embedded products are so flaky is just because of their shell scripts

The only thing that makes UNIX appealing to me is the fact that I find all the alternatives are so much more annoying. If there is some UNIX alternative that is less bloated and slow, please share.

NB. Almquist shell does not necessarily use readline. Neither NetBSD's sh nor Linux's dash use it. (Not to imply editline isn't bloated, too, but it's not nearly as bad.)

Further, I did not mean "the terminal". I had to look up "LARP". FWIW, I do not play video games. I do not use readline. I do not use a "terminal emulator". I use textmode. These choices are not because I think highly of ash, editline, textmode, non-graphical computing or minimalism in general. They are because I find the alternatives (large scripting languages, readline, video games, terminal emulators) are really annoying.

Why not share some of the programs H4ZB7 has written, so we can see how to do things the right way.

The shell is everywhere. The commenter trying criticise the shell is probably using it to launch the interpreter for their "proper scripting language" among other things.

In theory, the shell could be eliminated and people could boot into something else. Onine commenters love to bring this idea up. But in practice, this is rarely implemented. The UNIX shell is the overwhelmingly choice. This is not an "argument". It's a fact. The shell and shell scripting are everywhere.

The "proper scripting language", OTOH, is a highly opinionated concept. It will keep changing. A never-ending popularity contest.

Shell scripting is here to stay whether language wonks like it or not.

I don't recognize non-sequiturs and shifting the goalposts and as valid moves.

If don't like UNIX, stop using it. Let us know how that goes. It's all but impossible. Generally, one cannot even download or install a "proper scripting language" without using a computer that runs shell scripts.

UNIX has even crept into Windows. It began when they took TCP sockets from BSD. There was SFU. We have seen further development of a "subsystem". Today, Windows is constantly connecting to the mothership, and the mothership, like Azure, is running Linux. NB. There is nothing spectacular about UNIX. Windows however is unbearable. That's what makes UNIX appealing.

It's funny how this community trash on JS at every chance they have but for some reason bash is off the hook. I am tired on seeing CI pipelines failing and Live environment crashing because typos in .sh scripts. I propose to switch to powershell every time I can but that still sounds as heresy for the devops guys.

Lol... You are going to switch to PowerShell in initramfs?

PowerShell is just another Perl / Python / Ruby. Except made by Microsoft, so it's got a ton of useless features, lacking some essential stuff and has bombastic syntax. It's not suited for the role of UNIX Shell. Not by a long shot.

----

On a larger note, the whole idea of needing a shell is bad. It comes from a defective (or rather overly simplistic and un-insightful) design of UNIX. A better system wouldn't need a distinction between a system programming language and a shell. You would just use one and the same thing for both purposes.

If you are on Linux, Guile is supposed to be that, but... maybe it was supposed to be that?.. I'd still want it to take the role of Shell, even though it doesn't tick all my boxes.

Nobody said anything about switching to PowerShell at the lowest levels. But even though it is verbose, and has some annoying quirks (like over-eager unpacking of one-item arrays), it's much nicer to write scripts in. You don't need sed/awk to parse output of ls and hope it's stable, just get whatever data you need from the object. What essential stuff is missing?

> hope it's stable

Or worse, the output is stable (because peopleare scraping it with aws and sed) and reflects things as how they existed 20 years ago versus now. For example `ifconfig` doesn't really match how Linux does networking.

Ifconfig is not part of the shell. Whatever your grudges with ifconfig are, however justified, they have nothing to do with how the shell works.

BTW, I don't know of any distro wit LTS releases that didn't yet expire that doesn't ship with iproute2. In my opinion, iproute2 has done a very good job, given the circumstances to organize and systematize information about networking. So, if you don't like ifconfig, today you most likely have a better alternative.

Your comment seems to have f the point. I know ifconfig isn’t part of bash and I’m aware of ip addr but the point stands:

people don’t update tools because it breaks scraping.

If fact if they did ip addr wouldn’t exist: we’d be able to update ifconfig.

> people don’t update tools because it breaks scraping.

What people are you talking about?

Anyone who's on the payroll is liable if they don't upgrade systems when they go out of support. It doesn't happen because of someone's whims... If some individual user chooses to stick with an outdated system: it's on them, and they have no ground to stand on if they complain about it...

> we’d be able to update ifconfig.

No we wouldn't. "ip a" is a tiny fraction of what this command does. The reason it came to be was not that ifconfig needed replacement, but because Linux utilities are a zoo of halfbaked projects, most of which have huge flaws in their design, especially when it comes to extensibility.

Did ifconfig foresee bridge interfaces, bond interfaces, vLANs etc? What about IB? What about sr-iov etc.? -- Nope, and since there wasn't a single structured and systematic tool, that could've been extended by the authors of eg. bridge drivers, they were forced to come up with their own tools, eg. brctl. iproute2 wasn't an improvement on top of ifconfig. It was an aggregation and systematization of various tools and configurations that dealt in a very partial way with various aspects of networking.

There would've been no way to keep any kind of backwards compatibility with ifconfig, so there was no reason to keep the name. It's not an evolution of ifconfig, it's an entirely different thing.

> What people are you talking about?

Coreutils maintainers.

> Did ifconfig foresee bridge interfaces, bond interfaces, vLANs etc?

No. That's literally why I wrote the comment.

> It's not an evolution of ifconfig, it's an entirely different thing.

Yes, agreed completely.

But if we got away from scrapping we could have added them a lot more gently.

> There would've been no way to keep any kind of backwards compatibility with ifconfig

Disagree.

> there was no reason to keep the name

To this day people are still scraping ifconfig and wondering why the results are bad.

I've been writing Shell scripts for... well, over ten years at this point.

I don't believe I've ever parsed output from ls. Definitely not with sed or awk. I don't think that any Linux admin worth their salt would do that.

Needless to mention that neither ls nor sed nor awk have anything to do with Shell.

> What essential stuff is missing?

Since Microsoft tried to replace not just the shell, but the entire set of utilities with PowerShell, they forgot about xargs, for example.

But, if we are talking about the shell proper, then things that are missing would be a serialization format that would allow one to pretend that remote shell sends "objects" rather than strings.

Most importantly, however, PowerShell is an attempt to fix the problem at the wrong level. The reality of the OS it's running on is that processes take command line and environment variables, and they don't take objects. They return exit codes and two or more output streams, they don't return objects. PowerShell is trying to pretend that communication between processes happens in objects, but that's a lie, and it shows. In any non-trivial use of such a shell, you will have to escape the objects, and face the reality of what's happening underneath.

That's why earlier in the days, when objects were a novelty programmers thought that a programming language that is object-oriented from the bottom up is preferable to the one which implements objects as a library.

PowerShell is even worse in this sense than Perl or Common Lisp. Objects in PowerShell aren't an afterthought. The authors wanted them from the start, but couldn't have them. Unfortunately, we live in the world where most popular projects we use today were still-born in engineering terms (eg. Unix, and later Linux, any Fortran-like language created since 80's etc.) So, it may as well happen that PowerShell will succeed in terms of popularity, but it's broken by design unless we completely replace the whole way we work with processes... and I don't see it happening unless there's some cataclysmic event that wipes most of us out.

> Since Microsoft tried to replace not just the shell, but the entire set of utilities with PowerShell, they forgot about xargs, for example.

You can just use foreach with an array or a multi-line string:

$newFiles | foreach { git add $_ }

> Most importantly, however, PowerShell is an attempt to fix the problem at the wrong level. The reality of the OS it's running on is that processes take command line and environment variables, and they don't take objects. They return exit codes and two or more output streams, they don't return objects. PowerShell is trying to pretend that communication between processes happens in objects, but that's a lie, and it shows. In any non-trivial use of such a shell, you will have to escape the objects, and face the reality of what's happening underneath.

If you’re executing processes, then yes, things might get hairy. But if you’re using PowerShell cmdlets (which either do the thing natively, or call a subprocess and parse it to an object), the object-oriented stuff is really convenient and powerful.

Here’s a random example of fancy PowerShell things. How can you do this (filter by a keyword, group by parent directory, and sort the results by count) with UNIX tools? What magic combination of ls, find, sed, grep, awk, sort, uniq would solve this?

Get-ChildItem -Recurse C:\Stuff | Where-Object {$_.FullName -notmatch "xyz"} | Group-Object Directory | Sort-Object -Descending Count

> You can just use foreach

That's not what xargs does though...

> If you’re executing processes,

This is why shells exists. They don't exist to execute cmdlets. Nobody cares about that. Cmdlets are about internal functionality of the shell. Executing processes is the service shell provides to it's users.

> What magic ...?

I don't know what that code does. I don't have Windows, and wouldn't install Microsoft's products on my own or company's computer... But, my guess would be that it would take about half the effort that it took you to write this.

What magic to you is not magic to me, because I know that stuff. You must know PowerShell better than I do. I simply know enough not to use it. I don't care about the gory details.

You aren't going to write 100k lines of shell scripts but you might reach this amount of code in JS projects so we should rightfully hold the language to a higher standard. Shell scripts are just glue and unless they get huge they are easy to understand, modify and maintain.

I have a fair amount of shell scripts in CI pipelines and things usually break when we hit unhandled edge cases which is exactly what I would expect.

That said, if your script is for a single platform you won't need any of these hacks which only exist for compatibility with different/older shells. My scripts won't even run on bash 3 and I accept that.

Except they are not "easy to understand, modify and maintain" because of a bunch of weird bugs like the one from the article, and they don't depend on the script size

Whether or not a language is weird is just a function of how comfortable you are with the quirks. You could fill a textbook with just the unintuitive hacky shit you had/have to do in JS.

Esoteric languages (e.g. brainfuck, befunge) are evidence that language weirdness is somewhat measurable and objective.

PowerShell will happily run scripts with misspelled variables and not fail until it hits the invalid line, if it ever does. It's not like it's statically typed or something.

Most people don't like shell scripting, it's just that it's way easier to maintain the environment. A shell script from 20 years ago will likely still run without dealing with EOL runtimes or ensuring you're running an LTS version of a shell. The shell is quite stable from an API perspective, so you're not dealing with removal of deprecated features. Shell scripts largely run the same across different systems. With the notable exception of Apple shipping an ancient version of Bash, rarely do I have to care what version of a shell a system ships with. And that only matters if I want to use new Bash-specific features.

While shells can and do have security issues, much of that is historical and given the smaller surface area they just have fewer security issues than say, Node. I don't need a shell version manager à la NVM, virtualenv, or rbenv. I don't need to worry about a massive dependency graph. A shell of some sort comes pre-installed with every *nix system I've ever used so I don't need to worry about figuring out how to install a new runtime in an unprivileged environment.

I'm sure some people love Bash and love shell scripting. For many of us it's just a matter of practicality.

I am still waiting for a language that has variables, strings, loops and if-branches, and can pipe programs together.

OIL and ABS are close but who uses those.

The weird thing is we were already there in the early 90's with perl, and as a community uniformly decided to walk away from it. Now we're "still waiting" for stuff people were running on SPARCstations?

I think all but the die hards got tired of the jokes. If only they had held on a little longer, those would have stopped when people forgot about modems and line-noise...

Check out Tcl! It's built around strings, with a command syntax similar to shell. The standard library "exec" command allows launching programs including pipes and redirects. Although it's a bit clunky (e.g. there's no way to quote a '|' or '>' argument)

https://www.tcl-lang.org/man/tcl/TclCmd/exec.htm

Is tcl still mainstream? And what's the dependency install like on a standard system?

I used tcl 15 years ago and it was pretty great, but the SDKs were already getting old and many falling unmaintained. I worried about the future of tcl

On Fedora, I think it might even come installed by default (possibly as a dependency of Python's Tkinter module). There's also Jim Tcl, an embeddable implementation with very few dependencies.

pwsh. Also hashmaps (and structured data generally) so you can do

`ls | where createtime < now - hour`

That's pseudocode, I don't use pwsh every day.

pwsh never really took off as it was good tech during bad leadership (Steve Ballmer), but I hope someone (maybe Rust coreutils people) steal the approach.

pwsh never took off because it doesn't fit the spirit of unix system administration. pwsh is object based while on unix systems, it's all about shredding text. If you are on windows, use powershell as it makes your life easier. But I don't think anyone wants to use powershell on unix, linux, etc.

“Shredding text” (is scraping) is brittle. Nearly all programmers know this. Especially people that have been using Unix for more than twenty years.

Regarding you last sentence: yes that’s why I mentioned it didn’t take off in the comment you replied to.

But long term scraping will die.

AFAIK, perl still comes by default with all Unixes that are actually used.

But well, most languages support piping program together. You will need a 5 to 10 lines wrapper on the most used ones, but it's always well supported.

And as a bonus, you get to explicitly say how the pipe should behave, instead of searching the docs to learn how to handle failure or early stream termination.

Every major admin scripting language I've ever used has so many foibles and edge-cases it makes me pine for full programming languages.

What admin scripting languages bring to the table:

1) Easy access to executable binaries - no creating a Process object to describe it, just run it

2) Easy piping and filtering

3) A standard library designed for terse and easy-to-use access to administrative tasks.

4) Pleasant in REPL form.

That's the good side. The bad side is that they're all such a collection of hideous hacks and edge-cases for basic concepts like variable scope and typing that even the simplest operations that would be hilariously trivial in a real language are utterly agonizing.

There are many situations where you don't get to choose what scripting language you're going to use.

A posix shell script written in 1990s would still work today and likely will work 10 years from now (given that external commands it uses had not breaking changes). Not many proper scripting languages have such track record - most of them either had many small backward incompatible changes or an update which breaks many things at once (like Python 2 -> 3). Though I have some 20+ years old Perl scripts which still work today but it is hard to find a language which attracts more haters than Perl.

Discussed extensively last year: https://news.ycombinator.com/item?id=26776956

One key point there is that it’s still easy to trigger cases where you need this hack, because the cleverness is only stable for up to four arguments; so and, or, and parentheses all exceed the point of safety.

So keep doing it despite whomever it annoys. Also, throw some superfluous cats in to annoy them more.

All the cleverness and precedence rules in the article seem to ignore the fundamental problem: the test and [ binaries are regular programs, and they cannot distinguish between an argument that looks like “(“ and (. In both cases, they get the string “(“ in argv.

So x serves as a form of quoting. If I were to potentially invoke a binary to evaluate an expression, I would quote any string-like variable (using the x hack) and I would do my best to avoid numeric comparisons.

(Or use built-in shell features where the shell knows which values are quoted.)

> All the cleverness and precedence rules in the article seem to ignore the fundamental problem: the test and [ binaries are regular programs, and they cannot distinguish between an argument that looks like “(“ and (. In both cases, they get the string “(“ in argv.

Regular programs can make that distinction with no problem, the mistake is thinking that the outer quotes are part of the argument. In the context of the POSIX style shell, they're not, they signify that what is enclosed is one argument. So you'll have to use one of the various mechanisms the shell provides to indicate that you want to pass the quotes as an argument.

I'm not the person you replied to, but I was unaware of the fact that it was possible to tell the difference between the arguments ( and "(" in a regular program. May I ask how? My understanding was that the shell stripped the quotes before it ever reached the program (and that the only method of bypassing that would have been to escape the quotes or enclose the argument in single quotes, for example).

  $ echo "(" '"("'
  ( "("

"(" the argument is (

''"("' the argument is "("

Ah yes, that's what I meant by putting the argument in single quotes.

I thought you were saying that the statement 'In both cases, they get the string “(“ in argv.' wasn't necessarily true. My apologies!

> the mistake is thinking that the outer quotes are part of the argument.

> you want to pass the quotes as an argument.

Yes, that's the point - in the context of /bin/[, 'x' is the quote; that's how you quote string arguments to [.

I don't get what you're saying. x isn't the quote, it's made to be part of the string being matched, to catch edge cases. You don't need to understand quoting rules to pass 'x' as an argument as you would '"("'.

No need for that.

In shell the preferred form for "and" and "or" is outside the test, and you always quote anything with a variable reference, like this:

    if [ "$x" = "foo" ] && [ -z "$empty" ] ; then
      echo "Worked"
    fi

I think the "&&" is easier to read, as well as completely eliminating the problem.

That was also discussed in the thread: https://news.ycombinator.com/item?id=26778153

Not to mention this syntax short-circuits, which is usually desirable.

Oh, shell pet peev #301;

shell has "case" as part of the language, and case is way better at string evaluation that the "test" built in

case "$x" in (foo) echo Worked ;; esac

Sorry I should have elaborated...

"test" or it's other name "[" is a program or a built in. It takes a parameter to tell it what to do. It can evaluate strings, but it can also look at the file system or do a little numeric evaluation. Newer versions in gnuland may even be able to tell you if a unicode glyph is left to right or right to left. It does all this using - or -- prefixed options.

test -f /dev/sda will fail (it's a block device not a file)

test -f /dev will fail (it's a directory)

test -h /usr/bin/[ will succeed on some unices (it's a symbolic link to /usr/bin/test)

So -- what does [ "$x" = foo ] (or, more clearly, test "$x" = foo) do if the variable x happens to have "-f" in it?

The shell will interpret the " and the $x and hand to the builtin this:

[ -x = foo ] (or test -x = foo)

Now, the human is waving their hands "that's not what I meant no fair!" but a contract's a contract -- test might look around for a directory entry in the current directory and not find a file named =, or hey, it might, because = is a fair directory entry name. It may then poke at that associated inode and find that = is in fact a file, and return a 0 in the exit status. Or it may say "hey, file existence testing has one paramater and you gave me a filename, =, and another file, foo, what gives?

Regardless -- you've got no control over the contents of variable x; that's why you're asking test to look into it for you. You can blorp a thing in front of it to protect yourself from this case, but that's just silly. Test can't do any regular expressions and it does all sorts of other things, so it's taking parameters, it's just a mess. Just use case $x in ... esac.

Test is great for other stuff, but string evaluation? It wasn't ever good for it.

Case is better at pure string comparisons, but I believe the -x example is wrong. Test is PRIMARILY influenced by the argument count, so [ "$x" = "value" ] always does a string comparison for equality, never anything else.

Guilty here, but have been doing so for so long that it's largely muscle memory at this point and the original rationale has been lost. In mitigation, it certainly used to be the case that you were likely to come across a broad variety of Unixy flavours and hence using highly conservative scripting constructs made sense; then after a while these become second nature, and unless they cause actual problems they just stay in the mental toolkit.

Same here!

When I started off in the UNIX world, the shop had a collection of AIX, SunOS, Solaris, HP/UX and Digital UNIX. the x-hack was standard form and I spent a few years fixing errors with the x-hack (or converting the scripts to Perl).

It is just second nature to use the x-hack in if's and case statements on this point. I have to try really hard to NOT use it. Of course, the folks with less gray in their hair and often confused by it, referring to it a "Olde Unix."

I have approved plenty of PRs with the comment, "Updating mek's Olde Unix syntax."

Oh I will reference this from https://www.pixelbeat.org/programming/shell_script_mistakes.... where I also advise against this common archaic idiom

Wait... zsh, the advanced and powerful shell that's now by default on macOS too, would say "(" is equal to ")"?

Wouldn't be surprised if it was just backward compatibility so bash/sh scripts work

Using [ is a mistake. Use [[ and relax!

Unportable

Portability isn't binary. The question is, is it portable to the systems you want to support?

It's portable until it pages out on an incident report. Then, we know it wasn't portable.

> The question is, is it portable to the systems you want to support?

"Want" isn't always the whole equation. Sysadmins are often called to support ancient OSes, in particular in environments which keep systems in operation long after their "best before" dates (e.g. internal IT in banking systems).

There's literally no disadvantage, beyond aesthetics, to _not_ using the x-hack. For many of us older folks, using it is built into our muscle memory.

We need to support the most common OS in the world, debian. default shell: dash

bashisms are forbidden in every corner of the world, sorry

https://askubuntu.com/questions/1277922/what-are-syntax-diff...

By default, dash provides /bin/sh on Debian. But that doesn't prevent people targeting /bin/bash instead.

I'll also note that BusyBox supports [[.

Who is this "we"

Developers

bash is Essential: yes in Debian. /bin/bash is available on every Debian system.

But not in POSIX, and not in FreeBSD. There it is at /usr/local/bin/bash if you install it.

Only as unportable as bash. Which is probably the most portable shell.

You'd never do [ "x$left" = "x$right" ], but [ x$left = x$right ]. The whole reason for the x-hack was to omit the double-quotes. With the dquotes you can of course leave the x out. [ "$left" = "$right" ]

No you can't. Read the post.

What happens if you use the x hack but $left has a space in it?

Then you get a nice syntax error, that you didn't properly split your string. Good point

You won't get a syntax error if it happens to result in valid syntax (but likely unintended semantics).

Remember though, just because the shell on your rolling-release dev machine fixed this issue years ago, does not mean the shells on all your users/clients/legacy servers/hobby backports have fixed the issue too.

>However, the value was mostly gone by the mid-to-late 1990s, and the few remaining issues were cleaned up before 2010 — shockingly late, but still over a decade ago.

Anyone running a shell that still requires the x-hack is deliberately maintaining a legacy system, at which point the onus is on them to get modern tooling working on their machine.

Being forgiving towards the user is one of the important principles of programming. Because the user may not be able to “upgrade” his system.

Windows 95 is still very much alive.

Embedded-like systems are a thing.

Yet asking for windows 95 support on most open source bug trackers will get a reaction between being laughed off the tracker or "do it yourself, but we'll reject the PR if it makes too much of a mess". I don't see why legacy commercial Unix deserves any higher level of support these days.

Did not say so. Only that if one does do it, one is being understanding.

Did not say it has to be done.

Just because some projects (OSS or not) decide on a certain policy doesn't mean that the policy is necessarily good, or that any other projects should follow suit. Whether or not this is a reasonable stance depends on the project and the target demographic.

> Being forgiving towards the user is one of the important principles of programming. Because the user may not be able to “upgrade” his system.

Yes. If there is one thing I wish I could drill into every programmer's head, it's that not breaking stuff for your users is the #1 most important thing. There is one of you, and there may be many thousands of users. You putting in a little extra effort means saving thousands of times of effort from your downstream users. Sometimes breakage unavoidable, but those instances should be rare, extremely well communicated, and felt as a failure to be learned from and avoided in the future. I don't care how ugly it makes things for you, your users are paramount.

This isn’t even a moral principal - design systems and projects according to this discipline and you yourself will be repaid in extra robustness and less work in the long run, as well as happier users. Plus it is fun coming up with work around for obscure corner cases.

> not breaking stuff for your users is the #1 most important thing

I would say that's #2. #1 is to never cause data loss.

this is a perfect example of how UN*Xoids like to enamor themselves that they are the last bastion of responsible devs maintaining the 30 year old system that was never good to begin with from breaking

Outside of very specific situations that you'll know if you're in them, when are you expecting someone to simultaneously (a) insist on running ancient software that they can't update to decades-old standards and (b) run your new software?

> Windows 95 is still very much alive.

Can you share any examples/use-cases? I'm not doubting you, I'm just very, very curious.

In 2011 a customer I worked with was using Windows 95 still in a test bed for their legacy equipment. The test software they had was written for it, and it still worked, so no need to go through the massive pain of updating. But even that was 12 years ago and we expected it to fall apart any day...

A friend of mine still helps out a number of local businesses running their old MS-DOS Point Of Sales terminal software, as well as old high end printing hardware at a print shop. It works, and replacements aren't worth the CapEx as they're doing just fine. Migration has real costs, like losing the customer database for a mechanic's garage or buying a new 6 digit printer for the print shop.

I get what you're saying, but the fix to support that old shell is literally two characters. If you don't want to support it, it would be good courtesy at least to print an error and abort, lest the buggy behaviour ends up damaging the user's system.

Agreed. Abort with a clear message on unsupported systems. In this case, non-POSIX shells.

The alternative is to bundle all the dependencies, including the shell interpreter into some maximally portable setup.

I think telling your customers who rely on older systems to go pound sand is a good way to lose customers. Not to mention a bit callous.

Sure but if those customers are 1% of your revenue and 30% of your tickets then they're customers you want to lose and it's not particularly callous to want to feed your family.

Sure, but that's not the case for every business.

I'm not saying that everyone needs to keep legacy support in mind. I'm saying that just taking the attitude of "it's old, too bad" is not good blanket practice. It all depends on what you're doing.

> it's not particularly callous to want to feed your family

The callous part is taking a disdainful attitude toward those customers.

> The callous part is taking a disdainful attitude toward those customers.

I guess I thought it was obvious that most of the the time someone doesn't want to support legacy it's for this reason, not because of disdain for them.

That's what disdain is though (in a very literal sense), not feeling that someone is worth your time or consideration.

Your disdain of legacy clients might be profitable or understandable or even morally just, but it's still disdain.

> disdain somebody/something to think that somebody/something does not deserve your respect or interest

OED. You can respect someone but know their business isn’t worth your effort.

I get the sense that a lot of words have become generally mapped to "good" and others generally mapped to "bad" and it becomes difficult to have a nuanced conversation, especially around the not-uncommon situation where you disagree with or even dislike someone without holding the opinion that they are a bad person.

> it's not particularly callous to want to feed your family

Callous is callous regardless of your motive. It might be justifiable to be callous if that's what your business circumstances demand, but that doesn't change the effect on those customers.

> it's not particularly callous to want to feed your family.

Jesus Christ, you just put x in front of the variable!

Weirdly tho the legacy business is often a lot higher than the growth business at any point in time. It is often the case that orgs won’t break down the costs this way, but if growth has a lot of potential in the future, legacy is paying off previous growth bets now. So don’t piss off half your profits because you are more focused on what to you is the bright shiny new thing that you have high hopes for.

Nonetheless, having read the fine article, I will stop using he x construction, which I find myself typing from time to time just out of habit.

The thing is, that's working like partially taking antibiotics. In B2B, you often end up with a customer on an old, customized system making 10% of your revenue. Dropping that often isn't an option.

Most recent MacOS is still shipped with bash 3.2.57 FWIF. Since default was changed to zsh they would likely never bump bash. So you never know.

That's still a 2014 release--definitely modern relative to the releases talked about in the article.

That was a licensing issue. It's pretty much guaranteed Apple will never ship a newer version of bash.

up until last year I was supporting a system for a client that would only run on Solaris 2.5.2 from 1996 or so. (finding working versions of the hardware was becoming _really_ hard). The system was retired last year, replaced by something that costs several million dollars, but until then I had no choice.

There are other things like that out there. I think I recall seeing something on HN fairly recently abut a controller for some CNC machine that was running an early version of MSDOS. Again, replacing with something more modern was very expensive. These things exist and the users aren't necessarily being wilfully malign in not upgrading

You really want to cater to people who chose to install Dash, and then not update it for 8 years?

Revenue has no bounds.

Well yes it does, when it is exceeded by the cost or hassle of earning it.

I won't support Windows 95 for a pittance

I'm not sure any dash has ever been broken in a way that would be worked around by the x-hack. This is an issue for, basically, pre-standard shells.

Don’t shoot the messenger, please. I never even heard of Dash before reading this article. I’m only quoting the article, which states that the last shell that needs the x-hack in some weird edge case is Dash and it got fixed in 2015.

Fair enough! I guess we just have to conclude that it's for very good reason that POSIX basically throws its hands up and only tries to comprehensively specify behavior without conjunctions.

Dash got fixed in 2009. It's zsh (which isn't even POSIX compliant by default) that didn't get fixed until 2015. And almost all zsh users are either powerusers who should know how to update their damn shell, or post-2019-mac users who should let Apple do it for them.

Yes it does. Sorry, I'm not supporting some ancient buggy AIX thing. People tying their shell scripts into knots using hacks like this are just doing performative compatibility nonsense. It hasn't mattered in the real world in a decade.

Also, this is hardly going to be the deal-breaker for running your software on such systems unless you're expressly writing for them. Pre-standard systems are exotic in ways that are going to break whatever you're trying to do in way more exciting ways!

> It hasn't mattered in the real world in a decade.

This is not true at all. There are plenty of such systems, run by major companies, in daily use. It isn't the most common case, but it certainly matters in important parts of the real world.

Are we talking about a real example here? I'm very sympathetic to backwards compatibility, to the point I still write sh compatible scripts unless I have a real need for `[[ ~= ]]`, but of the thousands of scripts I've written in the last 10 years, none of them would have any use outside of a modern distro. Adding the x-hack to every comparison seems utterly insane to me, exactly as GP said.

But like I said, I'm sympathetic, so I'd love to hear a case for why I should do it anyway

In my last position, I had to write new (and maintain old) shell scripts for these systems all the time, for Fortune 500 companies. Upgrading those systems was not in the realm of possibility for various reasons.

Because our product has to run on a wide variety of hardware, it was essential to maintain compatibility both for the crufty old shell implementations and the shiny new -- so best practice there was to develop the habit of writing shell code that could run across the eons, including the x-hack. Multimillion dollar contracts depended on it.

If your situation isn't like that, then I understand why this is a nonissue. But there are plenty of situations where such backward compatibility is essential. Not the most common thing, perhaps, but not as rare as some think.

I do think there is a tendency amongst some programmers to think that everyone is running stuff that is at least moderately modern. In the real world, this is simply not true.

would it be fair to summarize your position then as:

"It probably doesn't matter, but you will build habits and at some point you may wish you already had the habit in place"

?

If so, that's a reasonable point, thank you

Yes, I think that's a fair statement. I've developed many habits over the decades that, most of the time, don't really matter. But when they do, they matter a lot and I'm very glad I had them.

That is false. The only systems which have /bin/sh shells that have that sort of bug are proprietary Unixes, which also ship improved shells that better conform with POSIX, installed in a different location.

A much smarter thing is for the script to detect, at the top, that it's running on a crappy /bin/sh, find the better shell, and re-execute itself with that shell (setting some environment variable to prevent recursion).

Then the rest of the script doesn't have to use unusually stupefying coding techniques. Just the regular stupefying techniques.

IMO either write pure POSIX shell (no BASHisms or assuming external programs not guaranteed by POSIX exist, no assuming FHS paths are in use, etc) or require a specific shell and set of dependencies. Check for that at the start, exit with error if things aren't found.

Scripts and programs will always get run in unsupported (broken) environments. If your program can't exit with a clear error in such a situation, it's not very portable.

Personally I tend to use the multi-line nix-shell shebang:

#! /usr/bin/env nix-shell #! nix-shell -i SHELL -p DEPENDENCIES #! nix-shell -I nixpkgs=NIXPKGS_TARBALL_URL

If you want `bash` as your shell, then `bash` goes in the SHELL location and as one of the DEPENDENCIES. If you want GNU coreutils that goes in DEPENDENCIES. Etc. The last line is optional, it pins the versions of all the dependencies you want. This also works for Python scripts.

If you aren't quoting the left hand side, you still need it if the left side is an empty string:

  $ foo=""
  $ [ $foo = "" ]
  -sh: [: =: unary operator expected
  $ [ x$foo = "x" ]
  $ echo $?
  0

I'm not arguing that this is a good idea, merely observing that this case remains.

Empty strings had always been the case that I'd been taught required the x-hack.

Isn't this one of the things that bash [[ ]] explicitly fixes?

I hope not, because $empty is necessary when you actually want to elide a missing value, like Python's `filter`

What a horrible language:x

It's horrible but it's home for a lot of us.

more like holm for you.

The parent I think is getting two things mixed up.

    [[ $var == "" ]]

works as you describe and as expected. What the parent is talking about is when you have an invocation like

    my_program $opt
    my_program "$opt"

Which when empty will evaluate to

    my_program
    my_program ""

respectively.

Well, on the left side. On the right side, [[ still requires you to quote, otherwise:

    % bash -c 'a=xy; b="x*"; [[ $a == $b ]] && echo "$a == $b"'
    xy == x*

My hot take is that you should avoid using [[, preferring to use [, which at least pretends to work like a normal program that can be invoked using normal quoting rules—it just interprets a funky mini-language on its command-line arguments. [[ is some exotic thing that seems reasonable at first glance but follows no rules but its own and varies its behavior depending on syntactic nits that literally cannot possibly matter and aren't even distinguishable for normal programs.

And then just quote consistently, which is something you need know how to do with almost every other line of a shell script anyway!

> you should avoid using [[, preferring to use [

I tend to agree, and have taken this one step further, preferring the "test" command:

  if test "$a" = "foo" ; then

I do this for a few reasons:

- "Subsetting" your language can be beneficial: https://wiki.c2.com/?LanguageSubset

- "test" reinforces the notion that this is just the name of a program, not a syntax construct, which encourages using other programs as predicates, e.g. 'if which -s foo'.

- "man test" doesn't force you to wade through a 5,000 line man page.

Recent example: https://leopard.sh/leopard.sh

Wow haven’t seen a c2 link not from me in so long. So much good and interesting thought there.

Well techincally... the == and != operator matches the LHS against the pattern on the RHS.

So [[ $a == "$b" ]] works.

    $ shellcheck oops.sh
    
    In oops.sh line 4:
    [[ $a == $b ]]
             ^-- SC2053 (warning): Quote the right-hand side of == in [[ ]] to prevent glob matching.
    
    For more information:
      https://www.shellcheck.net/wiki/SC2053 -- Quote the right-hand side of == i...

Unlike the bugs that the x$var works around this is all in the manual!

https://www.gnu.org/software/bash/manual/bash.html#index-_00...

Why would someone want to intentionally write broken code and then hack around it, instead of the obvious correct thing?

There are lots of ways to hack around lots of ways to break code.

Because when we try something that works, we have a bias that it must be the right way to do it.

It's probably the biggest drawback to self-learning.

You don't need the x-hack for strings. In shell, a widely-recommended form for ANY expansion of a variable uses quoting unless you have special requirements. E.g., normally don't use $foo, use "$foo". Tools like shellcheck enforce this. Not ideal, but not a big deal either.

> in shell

This advice is highly dependent on specific choice of shell. This kind of quoting is unnecessary and discouraged in zsh, for example.

I mean for using POSIX standard (Bourne) shell mechanisms.

If you use nonstandard mechanisms, then whatever.

> You can still see this on macOS bash today:

  $ str="-e"
  $ [ \( ! "$str" \) ]  
  bash: [: `)' expected, found ]   # bash

> POSIX fixes all these ambiguities for up to 4 parameters

So... It's not fixed, and the hack is still needed?

To be fair, that's bash 3.2.57 which was released 2014-11-07.

Oldest bash after that I have available is cygwin bash 4.4.12 (2017-10-13) which doesn't have the issue.

Just install a newer bash using ports or homebrew and bob is your shell uncle.

If your script is going to be used by people on macs it’s probably easier to use the hack than ask them to update. Given all she’ll weirdness, I don’t even think this is the most offensive

Linux's not unix!

Legally, FreeBSD isn't Unix.

There is, however, Inspur K-UX, a Linux distro which is a Unix.

Here is something I wrote a while ago which may help provide some context and that explains the differences between test, [ and [[ : https://jmmv.dev/2020/03/test-bracket.html

Helpful, thanks!

It is for this sort of reason that I use Python instead of shell scripts.

Yes. No gotchas there.

    >>> foo = b'foo'
    >>> foo[0] == b'f'
    False

Don't get me wrong. I love Python, it has been my daily driver for 20+ years (and it pays my salary), but it does have its own set of cognitive burdens.

Also, shell scripts I wrote in the 90:s work today. Python scripts - not so much.

Shell scripting is fine, but some things are just hard.

So here is my attempt of writing another shell:

https://github.com/MoserMichael/jscriptparse

- it has a Repl

- if you write a shell command within a backtick operator then there is some tab completion.

- things like dealing with structured data in the form of json or yaml are simpler, compared to doing complicated stuff with jq or yq.

- the language is written on top of nodejs

Bash is the hardest language to write comparisons in if you're not doing it every day. The syntax is so unlike anything else.

> Bash is the hardest language to write comparisons in if you're not doing it every day. The syntax is so unlike anything else.

worse than brainfuck? I find that hard to believe...

on a serious note though, I don't agree. the syntax is not that different from the awk -> perl -> ruby heritage. Just a few constraints/conventions due to whitespace significance that other langs don't have.

I agree. There are genuinely weird things about bash that increases the difficulty of writing for it, but I never really counted comparisons as one of them.

> worse than brainfuck? I find that hard to believe...

Malbolge perhaps. Brainfuck is tedious, but it's a language of 8 symbols; there's nothing to forget. But doing comparisons in bash calls for a whole quickref of its own.

I only ever encounter this pattern in autotools-related shell code, stuff that ultimately ends up in configure scripts.

There it arguably makes sense to be so robust against old/broken shells, since the whole point is to successfully configure the build on a broad spectrum of environments.

Here is a question: what is the point of

   if test x$var = val ; ...

instead of using the [ command?

Some years ago, I checked: the [ alias for test was already in the Bourne shell in 1979.

Assume the year is 1999. What real portability problem do you solve by avoiding [ in favor of test?

Real meaning that the rest of the script, and the software stack the script is related to, runs on a system where [ wouldn't work.

I always use test:

- it looks like every other thing in sh (`[` is the outlier)

- it prevents confusion like putting `&&` inside the `[ ... ]` brackets (understandable that `[` works that way if you think about it, but it’s another way it’s an outlier)

- mildly discourages use of one particular bash-ism, `[[`, as it works slightly differently, and I prefer POSIX sh unless there is a good justification

- it doesn’t require a closing `]`

I don’t use it for what I think you mean by portability, that is, because `test` exists whereas `[` does not, or anything like that.

For all of the arcanities, things like this really make me admire the pragmatic attitude of the grey beards. Kind of like how the limited register count in early CPUs led to "of course" reusing one of your operand registers to store an instruction result.