(It is late, bash examples are approximate. Don't sweat the syntax.)
the_tool *.yaml
shopt -s nullglob
the_tool *.yaml
There weren't any YAML files, so argc was == 0, which the program considered an error. Mea culpea, perhaps.
AN_ARRAY=(*.yaml)
if [[ < length of the array is not 0 > ]]; then
the_tool "${AN_ARRAY[@]}"
fi
So finally the contorted,
AN_ARRAY=(*.yaml)
if [[ "${AN_ARRAY+x}" = x && < length is not 0 > ]]; then
the_tool "${AN_ARRAY[@]}"
fi
Use it on the command line in your day to day, fine; but if it's getting automated, jump to at least Python.
find . -maxdepth 1 -name '*.yaml' -print0 | xargs -r0 the_tool
If you have a problem that involves "run a command with a list of parameters but not if the list is empty" and you don't immediately think of `xargs -r`, then consider that it's because you don't understand the tools, not because the tools are bad.
find has -exec, xargs isn't necessary.
But it's like, I have to turn my head around and stop doing one thing (globs), and switch over to another thing (find, xargs). The language has baited me down one way of doing it, when I shouldn't be doing it that way. I'm well aware of xargs (but not the -r flag). I typically avoid xargs, as our code has to¹ work on macOS & Linux, and xargs is one of those utilities whose available options differ between the two for even simple use cases. (Such as here, for the "does it exec on no args?".)
But it still proves the point: the equivalent Python is just so much more straight-forward, easier to read, and not apt to shoot me at a moments notice, and doesn't have bugs. A reader only familiar with Python's syntax can read and understand Python, and can write Python that is correct. Not so with bash, you have to understand, "oh, no, globs will screw you here, you need this flag from this utility" — no, it's lunacy.
Like, I can go down the road you're suggesting (and in the past, I did) and keep learning the nooks and crannies and intricacies of bash, each one adding to the nightmare fuel that is my understanding of shell. I can keep tweaking the script around various failure modes, letting each line fail in prod 3 different ways. Or, I can just re-write it in Python, or Rust, or any other sane language and have it never fail again, for reasons that would also either apply to bash, or apply and have worse consequences than bash. (E.g., the Python would raise, bash would just plow on with it.)
Your response is exactly the same as my response before I saw the light on PHP, C, etc.: "you're holding it wrong". No, it's a bad tool that guiles the user into holding it wrong. There are better tools that are easier to hold.
¹b/c despite all work being done for Linux, devs get MBPs, b/c the company can't manage more than exactly 1 type of machine, for everyone.
Popen(["the_tool"] + list(Path().glob("*.yml")))While dethanatos's point about Bash's default behavior with a non-matching glob is on target, the remainder of the example demonstrates a problem in the_tool, which will need to be worked around regardless of how you invoke it.
No?
is this a cloud AMD64 opinion that ignores the 2 trillion devices which Gartner predicts will never run python?
I recently finished a ~600 lines ash script and the kids at work which just graduated complained that *"Y u no python?" For them it was difficult to understand because apparently I avoided cat/grep/dirname/basename/etc and used shell syntax wherever possible to avoid shelling out. (the thing I did would only run when system reaches a certain load)
Everyone forgot that the target hardware has only busybox (ash) and was massively resource constrained. My only options were to implement what I was doing as an eBPF, in C, or shell (ash). The shell script took 4 days (and was refactored/optimized several times within that period), the C (or eBPF) program would have taken ~10 days and the eBPF which is the best solution is unmaintainable once I leave the company.
from a performance perspective I take a bash script any time. Also python programs over time have a risk of eventually pulling in a dozen dependencies.
Plus, if you are running shell out of necessity on a constrained system, that doesn't mean shell deserves to be a programming language where there's actual choice.
how so? there is no python or perl or ruby etc on the target. what would have been a maintainable option other than /bin/sh ? C/C++ ?
> where there's actual choice.
again what choice? :) as I said there is no other solution than what I've given. the system literally doesn't have anything else regardless what "I think" it should support.
Typically, time spent interpreting control flow in a bash script is dwarfed by time spent in the programs called by the script.
> It is literally interpret by lines
It's just splitting lines into words to build up commands. It's not compiling C++.
> starting a whole process per if statements..
In the common pattern `if command1; then command2; fi`, if command1 is a builtin (e.g., `test` or `[`), then bash does not create a subprocess.
There are many ways to run a command on every .yaml file in a directory. Personally I like using xtrace. Something like:
echo cd the_directory > 1.sh
echo "test -f *.yaml" >> 1.sh
ls -1 *.yaml|sed 's/^/the_tool /' >> 1.sh
chmod +x 1.sh
less 1.sh
sh -ex 1.sh
if ls *.yaml > /dev/null; then
the_tool *.yaml
fiIs it really right to not process any files just because one of them just disappeared?
In general, situations like this suffer from race conditions and time-of-check/time-of-use problems.
No idea what you mean here, the glob can expand to one or many.
> In general, situations like this suffer from race conditions and time-of-check/time-of-use problems.
I mean, that's a problem in every version presented so far, one that I don't think can be fixed externally without something extra like hardlinking to a new directory, because "the_tool" apparently can't handle missing files and they can go missing when "the_tool" starts up. That's why I didn't even bother.
In fact, handling that types of errors is what bash is suitable for. "If file is readable, process it" is not an unusual construct. Please do take care however to avoid races if this directory is a spool directory that can be populated at any time, in which case it is useful that rename() is atomic, sometimes an additional working directory is required. Again, this is how the operating system is designed and nothing the shell can gloss over.
I have seen many problems from previously unknown race conditions when concurrency is increased. Scripts have to be kept composable and trivial to understand in order to guard against that.
To be honest, I would argue to remove any script that looks like the last example above. It is not clear what side effects of that bizarre array construct are useful to the system. Keep scripts as simple and readable as possible, a trivial truth perhaps, applicable to every language, but one that needs to be reiterated until the end of our profession.
It absolutely can, I have no idea where you’re getting this idea.
xonsh (https://xon.sh/)
Been using it for a few years now. It's worth it.
Good news, they did! Even better news: it doesn't require python, it's cross platform, and supports an easy-to-read verbose style for scripts but a quick-to-write terse style for interactive usage. It's an absolute joy to use for nested data structures, everything is an object and it has full native tab completion.
Unfortunately it came out of Microsoft so a lot of people haven't given it the time of day yet. It's honestly one of the joys of modern CLI usage though.
$> ll /bin/powershell
lrwxrwxrwx 1 root root 9 Mar 26 14:40 /bin/powershell -> /bin/dash
$> cat ~/pwr_to_the_people.sh
#!/bin/powershell
echo "Hello Powershell"
$> powershell ~/pwr_to_the_people.sh
Hello Powershell
I mean, I think there's something to that. Bash has a lot of arcane rules; I personally never write an `if` statement right, what with the `[[` vs `[` and having to surround the condition with whitespace. I find that bash scripts carry a million little gotchas that will cause variables to not evaluate/expand the way you'd expect.
Say what you will about Python, but it's much easier to reason about what a script will do as compared to bash (unless the bash is _extremely_ simplistic).
I've only kicked the tires on both, but plumbum:
https://plumbum.readthedocs.io/en/latest/
and sh: https://amoffat.github.io/sh/
both feel pretty good in that regard.
For more centralized workflows, I could see Python shell replacements being helpful. Especially project specific scripts where the overhead of managing a venv is negligible.
But realistically replacing the majority of bash scripts? I just don’t see it.
If not, there are copycats like Ruby and Python . . .
https://pypi.org/project/subb/
I wrote this to simplify just that for my own tools; the subprocess module that comes with the batteries has a very general interface, i think that it is a bit complex for a quick script.
My objective was to get an abstraction, for a one line process run and extraction of the result, similar to what we had in Perl5 with the system library function. https://perldoc.perl.org/functions/system
Also the shell is impractical, when it comes to slightly more complex programs. There is a limit on what you can do with pipes. Maybe that's the reason why perl is that flexible, as they tried to bridge both realms: Perl had to be useful as a replacement for the quick shell like script, and to be useful as a general purpose programming language.
This really just reads like someone who doesn't know shell and does know python, but thinks the problem is the tool. They continually describe shell with phrases like "obscure and difficult-to-predict" while talking through things that are either obvious with experience or from context, and then write something different in python (i.e. something you could write in BASH that does something different) that's not less obscure but is only using python obscurity.
> An example of a shell obscurity is the way the current working directory is set. The cd command is clear enough, but in the presence of sub-shells using (), can make it difficult to discern a stack of nested shell invocations and how the working directory changes when the sub-shells exit.
Er... yeah? So you expect PWD to be a global and it was actually a local?
Or more explicitly:
> One of the shell’s ickier features is that variables tend to be global. There are some exceptions and caveats, however, that lead to shell scripts that are broken or behave inconsistently.
That's not a shell feature, that's just how people frequently write it - you can make functions and declare your variables local if you want. I can easily write a python script with all my variables at the top, too.