News for hackers. Stuff that matters.
Re:Writing Safe Shell Scripts (2019) by mekster on Saturday February 01, 2020 @02:25PM (#22212619)
Make sure to have ShellCheck either integrated in your editor or run it before executing.

It really tells many of the rules you're supposed to abide by and helps you write cleaner shell script.

https://github.com/koalaman/shellcheck

Re:Writing Safe Shell Scripts (2019) by aidenn0 on Saturday February 01, 2020 @02:33PM (#22212665)
Shellcheck is amazingly impressive at catching issues with shell scripts. It makes it very hard to write a shell script that does the wrong thing.

Also, look at oilshell[1]; it is bash compatible out-of-the-box, but has several options to make it incompatible, but safer (e.g. no field splitting of parameter expansion by default, making quotes much less needed).

1: https://www.oilshell.org/

Re:Writing Safe Shell Scripts (2019) by chubot on Saturday February 01, 2020 @03:53PM (#22213155)
FWIW I think ShellCheck is great and the state of the art, but Oil is partly (negatively) inspired by ShellCheck :)

Somebody integrated ShellCheck into Google's code review system about four years ago, right before I left.

So the result was that every code review I sent with a shell script was filled with red squigglies -- "add double quotes here". Most code reviewers don't really know shell, but if they see red squigglies, they will say "fix this".

The double quotes are of course technically correct, but when you're writing a shell script in a subdir 8 levels deep in the repo that only operates on 3 files in that subdir, it's overkill and makes everything ugly.

In other words, it was the typical "static analysis has annoying false positives" issue.

Most of the shell scripts I write deal with "trusted" filenames -- i.e. those checked into the repo. If you want to deal with untrusted filenames, then there are a lot of ugly techniques to do that.

So Oil is basically meant to make the default thing the right thing, and the common thing the short thing (Huffman coding, as Larry Wall calls it).

Re:Writing Safe Shell Scripts (2019) by jonahx on Saturday February 01, 2020 @04:12PM (#22213267)
> The double quotes are of course technically correct, but when you're writing a shell script in a subdir 8 levels deep in the repo that only operates on 3 files in that subdir, it's overkill and makes everything ugly.

Once it becomes habit the cost is zero, though.

And the alternative you're implicitly suggesting is: "Follow shellcheck except you are allowed to make an expert judgement and ignore it when you know it won't be dangerous and have determined that the code is unlikely to change in the future so that this exception will be safe going forward too."

I think it's clear that this isn't a practical rule on a team with many people -- especially where they are not all experts in shell, which is 100% of teams of many people.

Making the experts "add those annoying double quotes they don't really need in this case" is the far lesser of the two sins.

Re:Writing Safe Shell Scripts (2019) by chubot on Saturday February 01, 2020 @06:22PM (#22213919)
Yeah I don't disagree with that. If you're working in a big group with some shell scripts, and want to get stuff done, use ShellCheck.

I just find it annoying, hence the new shell :)

There's a lot more to Oil, but that's definitely one of the surface annoyances I want to fix. Although zsh actually does fix that, and for some reason I've almost never seen a zsh script.

Re:Writing Safe Shell Scripts (2019) by fctorial on Saturday February 01, 2020 @10:44PM (#22215016)
Many oh-my-zsh plugins are just collection of functions, which I think could be considered scripts.

https://github.com/ohmyzsh/ohmyzsh/wiki/Plugins

Re:Writing Safe Shell Scripts (2019) by chubot on Sunday February 02, 2020 @12:52PM (#22218204)
Yes sorry I should have said "I rarely see a batch program in zsh". zsh is customized with zsh scripts for sure, and they are very extensive.

I guess it's because zsh is not POSIX compatible by default.

Re:Writing Safe Shell Scripts (2019) by acqq on Saturday February 01, 2020 @06:03PM (#22213825)
Exactly. If one "expert" is developing only for himself he can allow himself not to bother writing a script "clearly" (i.e. failing the programs that perform source checks).

But as soon as the script is something that should be potentially sooner or later maintained by anybody else than the original author (and in any successful project that's practically sure) the only right policy is to produce the source code that's obvious and passes the source checks.

An "expert" would have no problem to in no time adjust his three-line code to fulfill all the expectations of ShellCheck.

If it takes more than that, that only proves that the checks are actually beneficial and even more important than the "expert" tells to himself: it proves that writing that specific code "cleanly" is actually hard for everybody.

Re:Writing Safe Shell Scripts (2019) by 3xblah on Saturday February 01, 2020 @11:10PM (#22215087)
"The double quotes are of course technically correct, but when ... it's overkill and makes everyrthing ugly."

The only way you know this is being a person who routinely writes shell scripts.

There is no substitute for knowing what you're doing.

Try running shellcheck on configure scripts.

Or try this

    curl -4O https://ftp.netbsd.org/pub/NetBSD/NetBSD-release-8/src/build.sh
    shellcheck build.sh
Or try shellcheck on the build.sh from buildroot.org.

These scripts are "safe" enough for hundreds or thousands of competent users to be running them every day.

Re:Writing Safe Shell Scripts (2019) by yyx on Saturday February 01, 2020 @11:37PM (#22215164)
>all I get are false positives

double quotes, mixing string and array, unused variables. Why is this normal?

Re:Writing Safe Shell Scripts (2019) by viraptor on Saturday February 01, 2020 @09:25PM (#22214749)
> in the repo that only operates on 3 files in that subdir

It does today. Will it change tomorrow? Will someone rename one of the files to include a space? It sounds like people saying "I don't need to escape this value in a query, it comes from a static list and it's safe." Yes - it's safe right now...

Re:Writing Safe Shell Scripts (2019) by mehrdadn on Saturday February 01, 2020 @05:04PM (#22213561)
It's a burden to you only because you're used to not putting quotes in some cases. But to some of us who always put the quotes, it's not really about that at all, and it's not a subjective issue - missing them is literally wrong. It's like sprinkling random .split(" ") calls in your Python code for no reason, which you'd (hopefully?) never do. And when something is semantically so wrong, the elegance of its syntax compared to a correct solution becomes pretty much irrelevant.
Re:Writing Safe Shell Scripts (2019) by effie on Sunday February 02, 2020 @11:31AM (#22217713)
Quotes are just a tool, not a religious requisite. Chubot is talking from experience, and he is right. Putting quotes on all shell expressions is a burden, because it's more keystrokes, and more characters on the display which make the source on display disorienting. When you use shell every day, you know where the quotes are needed and where they are just a noise. Use a tool - when it's needed.
Re:Writing Safe Shell Scripts (2019) by mehrdadn on Sunday February 02, 2020 @12:45PM (#22218153)
Yes, of course they're tools, but "religious requisite" is a pretty disingenuous representation of the argument I'm making here? Something being a tool doesn't mean its use or lack thereof can't be incorrect. Bikes are tools too, but you don't ride them on the highway, right?

Quotes are tools that affect the correctness of your control flow quite vividly. Of course if there are multiple tools for solving the correctness issue you're welcome to choose any of them (and maybe the choice between the tools is something you can ascribe to religion), but that doesn't absolve you of the need to solve the issue itself. When the argument is that a piece of code is executing objectively wrong instructions (and ones that would be so obviously wrong in other languages!), that's pretty much the diametrical opposite of what you can casually dismiss as religious dogma...

Re:Writing Safe Shell Scripts (2019) by hkiely on Saturday February 01, 2020 @05:10PM (#22213593)
Which is best for simply catching typing errors? Shell check? I never saw the instillation instructions for oil.
Re:Writing Safe Shell Scripts (2019) by chubot on Saturday February 01, 2020 @08:09PM (#22214451)
The idea is that with Oil there will be better solutions to the problems that ShellCheck points out.

As mentioned, the default should be right in Oil. The defaults are wrong in shell, and ShellCheck is meant to alert you of that.

Oil should have a style linter / formatter, but if it's designed correctly it shouldn't need the analogue of ShellCheck, which is more about correctness.

Re:Writing Safe Shell Scripts (2019) by butterthebuddha on Saturday February 01, 2020 @03:23PM (#22213001)
I really want someone to integrate fish and oil, and call the project "fish-oil" or something along those lines.
Re:Writing Safe Shell Scripts (2019) by chubot on Saturday February 01, 2020 @03:47PM (#22213118)
(Oil author here) Yes I think that's a great idea, and I hope to release a "liboil" so that people can do that. Let a thousand flowers bloom, etc.

The most likely thing is that Oil is going to be a much better language this year (more expressive, with safety features, and it should be fast), but the interactive UI will still be more bash-like than fish-like:

https://www.oilshell.org/blog/2020/01/making-plans.html

So I encourage anyone who's interested in that to get involved now as the interactive shell is also a years-long effort :) There are many links on the home page about how to get involved, and I wrote many blog posts so people can understand how it works.

----

Here are a couple posts on why Oil is a good foundation for an interactive shell:

https://www.oilshell.org/blog/2020/01/history-and-completion...

https://www.oilshell.org/blog/2020/01/alias-and-prompt.html

Re:Writing Safe Shell Scripts (2019) by silly_giraffe on Saturday February 01, 2020 @04:38PM (#22213430)
I want to say that I am continually impressed at your long-term vision and commitment you have taken to the Oil project. Your posts are a treat to read, and the care you have given to the problem is appreciated. We need to stop paving cow-paths, and discard historical baggage if we are ever going to build robust systems.
Re:Writing Safe Shell Scripts (2019) by chubot on Saturday February 01, 2020 @06:32PM (#22213990)
Thank you!

Although I have to say I do feel the tradeoff right now between having a strong design / globally consistent code vs. allowing local variation / a lot of people to contribute quickly. That is, allow people to get in, add their useful patch, their useful bit of knowledge, and get out, without caring at all about the rest of the program.

I think Linux, git, and GNU are the prototypical examples of the latter. Those projects are too big for one person, so the code is set up in a way to allow hacking locally. They're also arguably a mess. Anyone who's ever written against the container APIs in Linux knows what I mean (namespaces, cgroups, etc.)

The git UI is another famous example, i.e. the plethora of commands and flags that make little sense. Empirically it seems that you can move faster if you disregard consistency and coherence.

The shell language of course grew like that over 50 years. Bourne shell is pretty consistent, but one thing I learned is that ksh added a lot of bash misfeatures (they're ksh-isms not bash-isms), and it's pretty bad.

But shell is actually a significantly smaller problem than git (after all bash is maintained more or less by one person). So it should be possible to fix it and redesign it. Unfortunately I'm not sure if the lessons generalize to bigger projects, but I would like it if they do.

-----

On a more practical note, feel free to send more people this way if you want to see the project succeed :) The design is there, and it will hold up, but it needs a bit more manpower, especially for things like the interactive shell.

Re:Writing Safe Shell Scripts (2019) by kortilla on Saturday February 01, 2020 @09:59PM (#22214864)
The trick is separating out external facing (api/ux) consistency from code style/data-structure/etc consistency. It doesn’t really matter if you don’t have coherency across the whole project for the latter because that can easily be swapped out later if it’s even a problem.

Most single-person project maintainers have trouble relaxing the latter and end up staying a single-person project because of it.

Re:Writing Safe Shell Scripts (2019) by sirtoffski on Saturday February 01, 2020 @07:29PM (#22214276)
Couldn’t agree more! I also really enjoy shfmt, integrating both ShellCheck and shfmt had really helped me to save time and write safer, readable scripts.

https://github.com/mvdan/sh

Re:Writing Safe Shell Scripts (2019) by rapfaria on Saturday February 01, 2020 @04:17PM (#22213296)
Using shellshock.net can actually teach you a bunch about scripting as well
Re:Writing Safe Shell Scripts (2019) by urda on Saturday February 01, 2020 @09:10PM (#22214690)
Quick reminder that ShellCheck is licensed under the strict GNU GPL 3.0 license. For many professionals your employer will often block / avoid GPL code, tools, and libraries.
Re:Writing Safe Shell Scripts (2019) by ahnick on Saturday February 01, 2020 @10:17PM (#22214921)
Sorry, but what employers block the use of GPL tools? Most employers that write proprietary software don't want GPL code in their code, but using a tool like shellcheck to perform static analysis on your code doesn't make that software that was analyzed now subject to the GPL, so why would they block it?
Re:Writing Safe Shell Scripts (2019) by bradknowles on Sunday February 02, 2020 @08:36PM (#22220759)
Apple is strongly anti-GPL. And they’re pretty open about that.

Most companies that are anti-GPL aren’t so open about it.

Re:Writing Safe Shell Scripts (2019) by ahnick on Sunday February 02, 2020 @11:55PM (#22221503)
I get that Apple doesn't want GPL code included in their software, but their engineers can still use GPL tools without fear of opening up their proprietary code. I know of plenty of companies where you can not INCLUDE GPL code in the software the company makes, but I don't know of any company that says you can not USE any GPL software to do your job. Those are two different things.
Re:Writing Safe Shell Scripts (2019) by bradknowles on Friday February 14, 2020 @02:50PM (#22329696)
For the six month contract I did at Apple, they were violently opposed to using any GPL software in any way, shape, or form.

Just because you can do something, doesn't necessarily mean you should. That rule applies to the implementation of corporate policies regarding licensing, too.

Re:Writing Safe Shell Scripts (2019) by filesystemdude on Saturday February 01, 2020 @09:16PM (#22214710)
> For many professionals your employer will often block / avoid GPL code, tools, and libraries.

Indeed, which is why Linux is never found in a professional setting. /s

Re:Writing Safe Shell Scripts (2019) by urda on Saturday February 01, 2020 @09:17PM (#22214719)
1) The sarcasm isn't appreciated here on HN.

2) Not everything in the FOSS world is GPL'd.

3) There are plenty of users who have this restriction in their work and it is a real restriction to be aware of when discussing tooling.

4) That's just how the code is licensed and is related to the topic at hand.

Re:Writing Safe Shell Scripts (2019) by urda on Saturday February 08, 2020 @02:54AM (#22274598)
I'm disappointed in you HN. The downvoting behavior over just discussing the facts of a codebase or industry is unacceptable and unbecoming. Your feelings of how software licensing should be does not matter here.