I’ve also been doing way more with google cloud shell. I was always such a point and click guy when it came to setting stuff up in GCP but automating and scripting with bash in cloud shell using ChatGPT is so much faster and repeatable.

Somewhere in the middle of every stack is a bunch of bash and makefiles. Don't replace that or avoid it. Embrace it. It's not perfect, but it's important to know.

Other things don’t: you need non-trivial bash.

I don’t mind if people do their heavy lifting in VSCode, I like the tool myself.

But if you can’t unfuck the little daemon that VSCode talks to?

You’re out of commission. You’re at the mercy of people who can.

I didn't claim there was no shell - I claimed that IPC isn't performed with pipes.

> software that doesn't involve shell at some level

which would seem to me to encompass more of the stack. Like... okay, Android IPC isn't pipe-based. Does that release anyone from touching shell? Anyone working on a ROM is going to get their hands dirty with the rest of the OS. And I struggle to believe that any app developer is avoiding it (it's always the build step that does you in). Approximately nobody is working on just IPC in isolation.

Just curious: where exactly is the shell in `cmake -GNinja`? Or is CMake not a build system hmmm? Nevermind that some people use bazel or meson or something ... other than shell...

Shell scripts in your build system are a code smell. In cpp projects it's covering up for people that don't how to use the actually correct tool for the job: the CMake scripting language.

Right above it; cmake replaces make, but something still runs it. And IME that something is usually either a developer running it from a shell (ideally eventually with another shell command wrapping it to automatically build on file changes) or a CI system running it which is always a massive pile of shell scripts.

To be fair, I don't doubt that if you really tried you could build a dev environment that used zero shell. But you would have to really try, and half of the things you had to build to make it work would inevitably converge to an informally-specified, bug-ridden, slow implementation of half of Common Lisp^w^wBASH.

That’s like saying that people use lotion because they don’t know how to correctly jerk off with sandpaper.

Bash is ubiquitous and can be used everywhere, CMake scripting language can be used only in CMake, guess how many people know one better than the other?

Sorry I wasn't aware bash added task graph features in some release?

Just curious: where exactly is the shell in `cmake -GNinja`? Or is CMake not a build system hmmm? Nevermind that some people use bazel or meson or something ... other than shell...

Shell scripts in your build system are a code smell. In cpp projects it's covering up for people that don't how to use the actually correct tool for the job: the CMake scripting language.

https://www.shellcheck.net/ is amazing to spot pitfalls.

See my list https://learnbyexample.github.io/curated_resources/linux_cli... for learning resources for CLI tools, scripting, etc.

Not that long ago, I needed to write a script that would grab the last command in a shell file that had a list of commands, add an argument or two to it, and run it. Which proceeded to barf because the command had quotes, for some of the arguments had necessary spaces in them, and I couldn't figure out the way to get the necessary string in a way that bash could properly execute the command. My eventual solution was to just use python's shlex module to parse the line and output an executable command (and remark that I should have just started the script in Python in the first place)...

It does that just fine, I don't have all the details obviously but it sounds like printf %q or ${arg@Q} is your friend here.

Right, how couldn’t they know to type actual gibberish to make it work.

If you’re doing stuff with something, why not read the manual os that thing? Or, at least, the section related to what you’re doing?

In a program with proper arrays, I can instead parse X to the strings ["echo", "A B"], and then run execve to run ["echo", "A B"] without any issues. And adding another argument to make ["echo", "A B", "C"] is damn trivial.

No, bash doesn't handle arguments with spaces just fine. If you do the basic, obvious way to do things, things break, and you're expected to know magic invocations to do the right thing to handle potential spaces.

>In a program with proper arrays, I can instead parse X to the strings

Well, if I were to write it in Java or Javascript, both of which have excellent support for arrays (the shell does as well, actually - via "set"), it still wouldn't get me an inch closer to achieving the goal. It seems to me that the real problem is lack of a first-class shell grammar parser, which affects most languages, not just the shell itself. Python just happens to have a shell grammar parser that is shipped with the installation.

  # store the command in an array
  X=(echo "A B")
  # execute the command
  "${X[@]}"

  A B

  printf '(%s)\n' "one two" "three four five"
  echo "How you doin'?"

  while read -r line; do
    eval command=($line)
    command+=("arg with spaces" "other arg")
    "${command[@]}"
  done < lines.txt

  (one two)
  (three four five)
  (arg with spaces)
  (other arg)
  How you doin'? arg with spaces other arg

There's no requirement that a shell language suck as much as bash.

If you never run into this problem, fair enough, maybe you don't need bash.

A lot of us do.

There's also an LDP introduction from 20 years ago that is probably exactly as accurate today (https://tldp.org/LDP/Bash-Beginners-Guide/html/index.html) and an "Advanced" walkthrough (https://tldp.org/LDP/abs/html/)

bash script is "okay" I guess if your "script" is just a series of commands with no control flow.

Why do that when I can literally just ``` N=0 while read -r URL; do curl "$URL" | jq '.data[].someProp' | grep -v "filter" > "$N.data" N="$((N+1))" done < ./links.txt ```

The other thing is bash makes it exceptionally easier to integrate across different system tools. Need to grab something from a remote with `rsync`, edit some exif, upload it to a CDN? That's 3 commands in a row, versus god knows what libraries, objects, and methods you need to deal with in Python.

So that other people can read and modify this.

It's very easy to read and modify if you just write it out longwise, which is what I'd always do in some actual script. (I also like to put reading data at the beginning of the pipeline, so I'd use a useless use of cat here.) To illustrate:

  N=0
  cat links.txt \
    | while read -r URL; do
        curl "$URL" \
          | jq '.data[].someProp' \
          | grep -v "filter" \
          > "$N.data"
        N="$((N+1))"
      done

Then show how it is going to look like?

    N=0
    while read -r URL; do
      data="$(curl "$URL")" || { printf 'error fetching data\n' && exit 1 ; }
      prop="$(jq '.data[].someProp' <<< "$data" || { printf 'error parsing JSON response\n' && exit 1 ; }
      grep -v "filter" <<< "$prop" > "$N.data" || { printf 'error searching for filter text\n' && exit 1 ; }
      N="$((N+1))"
    done < ./links.txt

Edit: oh good, at least the formatting looks ok.

And yes, you can fix those pretty easily.. as long as you aware about them. But this is a great illustration why you have to be careful with bash: it's a 6-line program which already has 2 bugs which can cause data corruption. I fully agree with the other commenter: switch to python if you have any sort of complex code!

    set -euo pipefail # stop execution if anything fails
    cleanup () {
        rm $tmp_file
        popd
        # anything else you want to think of here to get back to the state of the
        # world before you ran the script
    }
    trap cleanup EXIT # no matter how the program exits, run that cleanup function.

    for N, url in enumerate(Path("links").read_text().splitlines()):
        resp = requests.get(url)
        resp.raise_for_status()
        prop = resp.json()["data"]["someProp"]
        matches = (line for line in prop.splitlines() if "filter" not in line)
        Path(f"{N}.data").write_text("\n".join(matches))

Chopping a script into a series of one-liners where every command in the pipeline but the first and/or last operate only on stdin and stdout, as far as is reasonable to do, is a great way to essentially write shell in an almost 'functional' style. It makes it easy to write scripts with minimal control flow and easy-to-locate effects.

Such scripts are ime generally much easier to read than most Python code, but I don't find Python especially easy to read.

    set -euo pipefail; N=0 while read -r URL; do curl "$URL" | jq '.data[].someProp' | grep -v "filter" > "$N.data" N="$((N+1))" done < ./links.txt

Also, I've been forced to work on a huge SCons based project and I guarantee python can make your life quite miserable when used for something it's not supposed to.

A lot of originally little automation/dev scripts bloat into more complicated things as edge cases are bolted on and bash scripts become abominations in these cases almost immediately.

I was looking up statically typed alternatives and stumbled upon Ammonite and Scala-CLI for scala. I haven't used them much, but Ammonite bundles some basic libraries including command line parsing, http, and json, which are probably 99% of what I used in Python too? And Scala seems like an interesting language too with decent editor integration.

What does pyright/pylance say?

It didn't help him. Which is a bit strange?

I use below, and only when necessary use ignore comment pragmas when third party libraries are not typed.

    #? message format settings
    show_column_numbers = true
    show_error_codes = true
    #? strictness settings
    disallow_any_unimported = true
    disallow_any_expr = true
    disallow_any_decorated = true
    disallow_any_explicit = true
    disallow_any_generics = true
    disallow_subclassing_any = true
    disallow_untyped_calls = true
    disallow_untyped_defs = true
    disallow_incomplete_defs = true
    disallow_untyped_decorators = true
    no_implicit_optional = true
    warn_redundant_casts = true
    warn_unused_ignores = true
    warn_return_any = true
    warn_unreachable = true
    strict_equality = true
    strict = true

If you can't do it in POSIX (sh and utilities) and don't want to do an extensive investigation of portability for what you need, pony up for Python/Tcl/Perl (all in MacOS base, by the way).

The same can be said of bash, especially since you mention Alpine (also FreeBSD)

Perl, though... ;)

(if Perl is not there it gets pulled in very quickly as a dependency of something, e.g typically you pull git, you get perl)

Used to be a viable strategy until they started to drop modules from the standard library at every single release.

That’s a bit of a ridiculous statement, there’s a small number of very-long deprecated modules removed in 3.12, and some more recently deprecated modules in 3.13. And these things are old, largely or completely unmaintained, and usually complete obsolete.

I’d be surprised if anyone has a script that’s been adversely effected by this, and if they did it’s because they stopped maintaining it years ago (and also chose to both silence warnings and upgrade interpreter versions without reading the release notes).

Consider that the python foundation absolutely has the resources to put a developer to maintain them.

If they don't is because they don't want to.

> and usually complete obsolete

The amount of modules I've had to patch to keep working on 3.12 tells me they aren't as obscure and unused as you think they are.

> I’d be surprised if anyone has a script that’s been adversely effected by this

I'd say that over 99.9999% of python users do not download python from python.org. They use whatever is on their system. Which means that updating an LTS distribution will create mayhem. And that's considering that most modules have already been patched by the distribution maintainers to fix all the brokenness introduced by the new python version.

Also, a bash script from 30 years ago still works fine. A python script from 5 years ago doesn't start.

The resources to pay someone doesn’t mean that someone with interest and knowledge exists, especially for modules that were formally deprecated in Python 2 and which will never be reinstated. Lots of this stuff is just cruft, most of which has an obvious replacement, and if it doesn’t there’s a decent chance it’s not been used in years by anyone and if it ever had a reason to be in the standard lib, that reason is long gone.

> The amount of modules I've had to patch to keep working on 3.12 tells me they aren't as obscure and unused as you think they are.

If that number is at all significant, where are the issues pushing back against deprecation and removal? It’s not like there hasn’t been a formal process for all these modules. What got deleted in 3.12 was well documented and easily caught just by catching DeprecationWarning… anyone getting surprised by these modules going missing isn’t doing due diligence.

> I'd say that over 99.9999% of python users do not download python from python.org. They use whatever is on their system. Which means that updating an LTS distribution will create mayhem.

And I’ll pretty much guarantee you that 99.9999% of those users haven’t heard of, much less imported, any of the modules that have been removed.

> And that's considering that most modules have already been patched by the distribution maintainers to fix all the brokenness introduced by the new python version.

But again where are the issues and hands being waved that these issues are wide-spread enough to halt or reverse the deprecation process? If distro maintainers are simply patching everything for users who are constantly advised to leave their system Python alone and they’re not reporting the issues then those distro maintainers are harming everyone.

> Also, a bash script from 30 years ago still works fine. A python script from 5 years ago doesn't start.

I’ve written plenty of Python scripts that are still running on the interpreter and stdlib they were authored for, decades later. I’m also keenly aware that most of those scripts could not be written in Bash without reimplementing a significant portion of the Python standard lib and ecosystem, none of which was materially affected by the 3.11>3.12 removals.

That's why you can pay people. So that despite their disinterest they will read the code and acquire the knowledge needed.

> especially for modules that were formally deprecated in Python 2

??? I'm talking about modules removed now, in 2024. They were not deprecated since python2. Please don't steer the conversation to different topics.

> Lots of this stuff is just cruft, most of which has an obvious replacement

distutils? Is it cruft? The thing used to install modules? Can you tell me which stdlib replacement it has?

> it’s not been used in years by anyone

Why did I have to patch over 10 modules?

> and if it ever had a reason to be in the standard lib, that reason is long gone

Is installing modules no longer a thing then?

> those distro maintainers are harming everyone.

Aaah yes, the evil distro maintainers that keep things working instead of breaking everything. They're the real culprits here… really?

> I’ve written plenty of Python scripts that are still running on the interpreter and stdlib they were authored for, decades later.

Decades later? That's at least 20 years. If that were true they'd be written in python2 and I can promise you they wouldn't work with python 3.12. So I'll consider this statement a lie.

Please try to be more honest when you interact with me the next time. This hasn't been pleasant.

Bash scripts and Bash control flow has been and is used in highly critical scripts all over the place, including other planets.

We’ve been writing reliable, well-designed scripts for many decades. Some of my scripts are several hundred lines long and older than the system engineers currently occupying their positions.

Python is fine too. Use the right tool for the right job.

Instead they removed distutils, so now there is no way to install any module without using a 3rd party installer.

What advantages does zx have over Python, Ruby or even Fish for example, if we're installing extra programming languages?

[^1]: https://github.com/dsherret/dax

There are many reasons to use either. Probably about as many as there are to use the other. Choose wisely.

That's a strawman though.

I don't want to have to install a "vast selection of third party modules" just so I can run `kubectl apply -f foobar/*` on my CI pipeline.

It seems to be quite fast, can escape UNIX-Linux idiosyncrasies and I still can see all the magic variables in the code after those years.

Maybe it’s better to just get back into Perl instead of figuring out how to pipe back to back to few applications and jq on top?

Python unfortunately doesn't shine where Perl is good at. Mostly of the command line work involves passing around adhoc text, parsing and using it.

You are better off using a language designed to do that very task than use something else.

A bash script is basically scripting what you do on your terminal. I feel like scripting in the same language that is your terminal interface is underrated.

You can basically take a look at your command history and wrap it up as a script.

Yes, bash is clunky as hell. It has some really ugly warts. But learning the basics will make you more proficient in the terminal. The synergies are really nice.

Ruby's https://github.com/ruby/shell (which used to be bundled in Ruby itself) doesn't get much mention but is much more clear and useful to me.

    # system commands are not defined by default, only builtins, for safety
    Shell.def_system_command 'ls'
    Shell.def_system_command 'grep'
    Shell.def_system_command 'jq'

    Shell.new.transact do
      # simple redirs just work
      ls | grep('test') > 'some/file'
      ls | grep('other') | tee('foo') >> 'some/file'
      
      # pops at the end of the block
      cd 'some/dir' do
        # nested redir for demo
        (cat < 'foo.json') | jq '.[] | { bar: .bar }'
      end
    end

https://pypi.org/project/plumbum/

(also supports dispatching commands to SSH sessions)

Also Ruby's Shell was part of the stdlib, so if a system had Ruby you could use it without having to go through system ruby `gem install` conundrums. I think they made a mistake unbundling that one.

Otherwise, shell scripting is nice

Example: pyinvoke/fabric https://www.fabfile.org

How do you debug it and trace it ...

I do use it as said this is the terminal command line and both Mac/Linux supported it. And if it works, it work even nohup.

Just hard.

[0] https://www.shellcheck.net/

It also recommends wrapping variables with curly braces like ${name}, without explaining why, later on never does so itself throughout the rest of the book. There are a lot of such missing explanations in the whole book. It does make it short and fun to jump straight into the examples but i think it could benefit from some facts and reference material, at least to link to.

Anyway, seems like most people here are more interested in arguing over bash vs python than discussing the actual content. A little surprising to see this get so many points.