News for hackers. Stuff that matters.
Re:Common shell script mistakes (2008) by majestik on Saturday June 04, 2016 @01:43AM (#11835102)
Ok, I'll bite.

>> My personal belief is that everything that you can do in other scripting languages, you can also do in Bash, only better.

1) Native JSON, XML

2) Classes, namespacing, objects

3) Multiprocessing, multithreading

4) Performance

5) Package management

6) Portability

7) Documentation

8) Runtime debugging (!set -x)

I'm too tired to continue.

Re:Common shell script mistakes (2008) by int0x80 on Saturday June 04, 2016 @03:21AM (#11835350)
>3) Multiprocessing

IMO shell makes it very easy to work with multiple process (&). It's built in and natural.

>4) Performance

If you are carefull and know what you're doing, you can achive very good performance with the shell. Usually, better performance is achived processing less data, ie being inteligent. Rarely depends on the language (unless you care about cycle level performance, then yes :).

>6) Portability

I claim that it's way easier to depend on sh being on a (UNIX) system than $SCRIPTING_LANG.

>7) Documentation

?? You can mess up documentation in any language.

Re:Common shell script mistakes (2008) by jerf on Saturday June 04, 2016 @09:41AM (#11836279)
Shell makes it easy to spawn multiple processes. It makes it reasonably easy to read those processes' standard out or standard error, though it's not that much fun to try to do both at the same time while keeping them distinct. [1]

It pretty much doesn't do anything else that you might want to do with multiple processes, though, and it tends to encourage multiple processes to communicate via text which is a problematic limitation that one often finds oneself "working around".

Shell is really powerful, but it hits a certain limit of what kind of tasks it can do and it hits that limit hard, and that's why when one imagines orchestrating many processes on a machine to do some task, to say nothing of orchestrating many processes on many machines, you don't see solutions based on shell, and indeed the very idea is laughable. Shell is best used by making sure it stays firmly restricted to the domain it shines in and not so much as trying to dip a toe into the spaces where it is not.

[1]: Note "not much fun" != "can't". Shell is fundamentally written around the idea that a process has one stream STDOUT that may go to other processes, and one stream STDERR which is generally intended to go to the console (or other user output like a log) no matter how complicated the pipeline. While you can get both streams and do things to them, you're starting to fight shell, which really wants to create pipelines with one "through" path with no branches out.

Re:Common shell script mistakes (2008) by int0x80 on Saturday June 04, 2016 @02:03PM (#11837243)
I think with the shell you have to adapt your abstractions to the "unix-way". For example, a queue to process will be a directory with N files, and each file can be processed in pararell by just something like "for f in dir/*; do process.sh "$f" & done;" but yeah ... it has limitations like everything.
Re:Common shell script mistakes (2008) by CJefferson on Saturday June 04, 2016 @07:29AM (#11835938)
With regards (3), my problem in shell is that it is very hard to spawn children without risking overloading the machine.

What I would like in bash is some easy way to limit the number of background processes I can spawn, and to just wait when I try to start another one until an existing one is finished.

Some simple jobs can be converted to use xargs -P, but for more complex things I end up having to do them without parallelisation, so I don't end up spawning 100s of background processes and bring my computer to it's knees.

Re:Common shell script mistakes (2008) by int0x80 on Saturday June 04, 2016 @02:11PM (#11837280)
Yes ... I think that should not be allowed (bring down the machine by a non-root user process). In Linux CPU_GROUPS/MEM_GROUPS can help, and the fair scheduler has improved the situation a bit from the old days where a fork bomb will bring the machine down.

But limiting the # of spawned children is possible using not so complicated ad-hoc solutions, but I guess it depends on the specific problem.

Re:Common shell script mistakes (2008) by pwd_mkdb on Saturday June 04, 2016 @02:03AM (#11835156)
my personal belief is that anything one can do in bash i can do in sh. not sure if that's really true in practice, but that's my belief. i never use bashisms because i do not know what they are or how to use them.
Re:Common shell script mistakes (2008) by brbsix on Saturday June 04, 2016 @06:07AM (#11835741)
You're just having a laugh at the OP, right?

In case you're not, here are some "Bashisms" that really suck to be without:

* built-in regex support (e.g. `[[ $var =~ ^1\.2\.[34]$ ]]`)

* process substitution (e.g. `diff <(before_command) <(after_command)`) and all sorts of other redirection tricks

* indexed and associative arrays

Some of this can be worked around by shelling off to grep for regular expression matching or awk for arrays, but Bash makes things so much cleaner and maintainable.

Re:Common shell script mistakes (2008) by the_why_of_y on Sunday June 05, 2016 @07:49AM (#11840495)
Add to that better error handling.

AFAIK in POSIX /bin/sh it's not possible to detect if a process that writes into a pipe exits with an error status.

bash has "set -o pipefail" and "$PIPESTATUS" for that.

Re:Common shell script mistakes (2008) by cm3 on Saturday June 04, 2016 @08:18AM (#11836060)
But once you adopt bash-only features, you're losing the main argument for a shell script: portable scripting without the need to first install something to get something else running. Once you require Bash, it's equally easy to demand Perl and that will provide a much richer scripting experience.
Re:Common shell script mistakes (2008) by bogomipz on Saturday June 04, 2016 @09:23AM (#11836220)
When it this last time you logged into a box and /bin/bash wasn't an option? Let me guess 1999 on a SPARC box running Sun Solaris?
Re:Common shell script mistakes (2008) by cm3 on Saturday June 04, 2016 @10:16AM (#11836399)
If you limit yourself to either a popular Linux distro or one of two Unixes/Unix-likes, then bash can be available out of the box. Just as C is not C++, shell is not bash, so a shell script is what runs on (d)ash, busybox, toybox, ksh, bash, zsh, etc., without modifications. If it requires bash or zsh, then it's not a shell script but a bash/zsh/fish script. To name a popular non-Linux OS, take a fresh FreeBSD or OpenBSD install. No bash to be found, unless installed via ports and rightfully so. That said, I use bash myself all the time as an interactive shell but /bin/sh is not bash. sh (including bash and zsh) are terribly hard to write correct and resilient scripts in, and even rc is much saner to script in.
Re:Common shell script mistakes (2008) by 0xcde4c3db on Saturday June 04, 2016 @12:15PM (#11836815)
> To name a popular non-Linux OS, take a fresh FreeBSD or OpenBSD install. No bash to be found, unless installed via ports and rightfully so.

And even if you do install it, it still won't work with scripts that specify /bin/bash (which is a lot of them, thanks to sloppy tutorials), since it will be in /usr/local/bin and not /bin.

Re:Common shell script mistakes (2008) by JdeBP on Saturday June 04, 2016 @09:06PM (#11839118)
> When it this last time you logged into a box and /bin/bash wasn't an option?

A few hours ago; and before that, last Wednesday. The latter says:

    JdeBP % /bin/bash
    zsh: no such file or directory: /bin/bash
    JdeBP %
The Z Shell is an add-on here, too. /bin/sh is the Korn shell. No, it's not OpenBSD.
Re:Common shell script mistakes (2008) by ta0967 on Saturday June 04, 2016 @06:11PM (#11838374)
today. the various BSD systems all come without bash in their default configurations (and it's never in /bin).

this should not be construed as an argument in support of /bin/sh.

Re:Common shell script mistakes (2008) by jschwartzi on Saturday June 04, 2016 @12:16PM (#11836822)
On QNX all you get is ksh, which has little support for bash-isms.
Re:Common shell script mistakes (2008) by drdaeman on Saturday June 04, 2016 @12:20PM (#11836840)
Do embedded systems count? ;)
Re:Common shell script mistakes (2008) by bogomipz on Saturday June 04, 2016 @08:20PM (#11838958)
I'm having trouble remembering the last time I saw a BSD box during my work day, unless Juniper gear counts. No knock against BSD but seriously how many people are writing POSIX shell scripts because they need their shell scripts to work with Linux, QNX and BSD?
Re:Common shell script mistakes (2008) by JdeBP on Saturday June 04, 2016 @08:51PM (#11839069)
You should be first asking how many people are writing POSIX shell scripts because they need their shell scripts to work with Debian and Ubuntu.

* http://unix.stackexchange.com/a/250965/5132

* https://debian.org/doc/debian-policy/ch-files.html#s-scripts

* http://manpages.ubuntu.com/manpages/xenial/en/man1/checkbash...

* http://manpages.ubuntu.com/manpages/xenial/en/man1/posh.1.ht...

The answer is, of course, lots of them; as there was a massive project to do exactly this in Debian and Ubuntu some years ago.

Re:Common shell script mistakes (2008) by drdaeman on Sunday June 05, 2016 @10:37AM (#11841170)
A few, but there are still cases where it's best to stick with the plain old Bourne shell syntax.

When I mentioned embedded systems I primarily thought of Linux/Busybox-based devices, like OpenWRT. While one surely can have bash there, usually base image doesn't contain it.

Same story about the most common Linux-based OS out there: Android. And, while it's no one manually runs shell scripts there, apps quite frequently exec() things, so shell scripting still matters there - a tiny bit. Also, shell scripting is heavily used in firmware upgrades/patches, as well as a glue for the root/unlocking hacks.

If you plan to write a redistributable component or an useful hack for such platforms, you'd best stick with a very limited subset of POSIX.