News for hackers. Stuff that matters.
Re:What I learned from others' shell scripts by praptak on Wednesday August 14, 2013 @03:27AM (#6210290)
Shell scripts have their use but at the point when your script needs output coloring or printing debug information, it is time to switch to Python.

Edit: Place a general disclaimer to weaken the absolute tone of the above statement. Rule of thumb, exceptions apply, yadda, yadda. In other words, I somewhat agree with most responses that disagreed with the statement above :-)

Re:What I learned from others' shell scripts by 616c on Wednesday August 14, 2013 @03:52AM (#6210346)
I get what you mean, but there are reasons for shell scripts to stay shell scripts, and not move to Python. One such project [0] is patching Android to allow for a privacy framework. What would Python offer me for diffing and patching that cleanly written shell scripts would not?

Shell is still good for one thing: piping commands together. You can do that in Python, but I think even complicated output coloring and debugging scripts should stick to shell when the focus is piping utilities in and out of each other. Python can do that, but it is not its sweet spot, in my opinion. I am sure others disagree.

Different tools for different jobs, blah blah.

[0] https://github.com/mateor/auto-patcher

Re:What I learned from others' shell scripts by praptak on Wednesday August 14, 2013 @04:58AM (#6210511)
Agreed. Piping and a few of the other syntax niceties ('&&' for conditionally chaining processes is my favorite) are a hard match for even the best process-management library you might have in a general purpose language. Maybe a Lisp DSL could match that, I don't know.
Re:What I learned from others' shell scripts by klibertp on Wednesday August 14, 2013 @07:24AM (#6210878)
Check it out and say for yourself - I didn't try it yet, but as a Lisp fan I certainly will:

    http://www.scsh.net/docu/docu.html

    http://www.scsh.net/docu/html/man-Z-H-3.html#node_chap_2
I especially love the acknowledgements section :)

   http://www.scsh.net/docu/html/man.html
Re:What I learned from others' shell scripts by 616c on Wednesday August 14, 2013 @08:25AM (#6211072)
Does it still work well? I am very interested in this. I really like Chicken Scheme because it compiles to portable C, and I was wondering why there has not yet been a Scheme shell this morning.

Guess I was naive to think there was not without a proper Google-fu session.

UPDATE: I take that back, it appears some still works on it. That page just seems outdated, by a while now.

https://github.com/scheme/scsh/commits/master

Re:What I learned from others' shell scripts by jboynyc on Wednesday August 14, 2013 @08:02PM (#6215315)
As a Chicken Scheme user you might also appreciate the Hintz Chicken Scheme Shell (Hiss): http://thintz.com/chicken-scheme-shell

Disclaimer: I haven't used it and the author says it is "still pretty new." However, it seems to be under active development still. At least there was a commit to the repository within the last year.

Re:What I learned from others' shell scripts by 616c on Thursday August 15, 2013 @06:27AM (#6217011)
Very cool. I will check it out. Have an upvote!
Re:What I learned from others' shell scripts by noonespecial on Wednesday August 14, 2013 @05:57AM (#6210652)
I use these languages in ascending order for sys-admin/config:

shell -> perl -> python.

On OpenWRT systems, sometimes even microperl won't fit. All you've got is shell. (and sh only, none of this fancy bash stuff(1)) Its good to be able to do things at many levels.

(1) http://stackoverflow.com/questions/5725296/difference-betwee...

Re:What I learned from others' shell scripts by Torgo on Wednesday August 14, 2013 @12:50PM (#6212616)
At my first "real" job I used to work on a RS-6000, a couple hundred-grand machine. The admin wouldn't install _anything_ non-stock other than a JVM on it. It taught me how to get around using only BSD Unix tools and Bourne Shell. It was an enormous pain in the butt, but what I learned has been utterly invaluable over the course of my career.
Re:What I learned from others' shell scripts by reidrac on Wednesday August 14, 2013 @02:13PM (#6213213)
Well, it's more like: shell -> awk -> perl -> python.

I use awk from time to time, specially when I need to process a file line by line and the task is too simple to use python (ie. an awk one-liner will do!).

Re:What I learned from others' shell scripts by noonespecial on Wednesday August 14, 2013 @03:24PM (#6213640)
Very true. I really need to sharpen my awk/sed/grep skills, but damn, some of those 400 character awk "one-liners" hurt my brain.

Perl regexes are rightfully known for being cryptic but those awk statements make me cry for my mama.

Re:What I learned from others' shell scripts by mhd on Wednesday August 14, 2013 @04:25AM (#6210439)
Or Perl, of course. Although I've seen some pretty big collections of shell scripts used for deployment, configuration etc, and all that in a rather heterogeneous Unix environment. Then again, that was Korn Shell, which supports larger programs a bit better than plain bash.
Re:What I learned from others' shell scripts by cbsmith on Wednesday August 14, 2013 @04:30AM (#6210457)
Korn shell really ought to be considered "Bourne shell compatible Perl". ;-)
Re:What I learned from others' shell scripts by unhammer on Wednesday August 14, 2013 @06:36AM (#6210743)
I tend to think of colouring and debug output as standard shell script boilerplate.

I typically switch to Python (or similar) when I need something like multidimensional arrays / dicts (not portable in awk), or in general want to run more complex parsing or transformations on input. The more advanced string functions of bash I always have to look up (is it ## or %% that removes from the start of the string?), and for/while loops are in general a lot slower.

But for file system operations, stringing together commands, keeping track of backgrounded commands (using multiple threads without even thinking about it), and even turning arbitrary programs into "daemons" (using fifo's), nothing beats shell scripting.

Re:What I learned from others' shell scripts by jzwinck on Wednesday August 14, 2013 @08:30AM (#6211086)
You say nothing beats shell scripting for:

- Filesystem operations. How many times have we seen programs using "ls * .foo" when they needed to use "find -name '*.foo'" to avoid command-line length limits? I answered this very question on StackOverflow this week. How many seemingly-capable software shops will churn out shell scripts which misbehave when a path has a space or other "weird" character in it? Even Apple fell prey to that one, about ten years ago, and destroyed some users' data.

- Stringing together commands. This encourages abominations like "cat myfile | grep foo | awk '...'". Just use awk if you're into that, but the shell has a knack for "tricking" people into spawning extra processes that are not really needed (indeed this is one of the most frequent performance sinks in shell scripts). And what about error handling for the several subprocesses? It's usually ignored for N-1 of them.

- Keeping track of backgrounded commands (using multiple threads without even thinking about it). Yes, you can use multiple cores without thinking--that can be cool. But what if you want to do N units of work on many fewer than N cores? You ought to use a pool, but there's no such thing in Bash. Maybe you're clever and use "xargs -P" for this, but most people don't.

- Turning arbitrary programs into "daemons". I use start-stop-daemon for that (it's included in Debian, and I easily wrote a workalike in Python when I had to use a system that didn't support it natively).

Just about the only thing here that shell scripts are really good for is doing things "without even thinking about it." Once when I was asked why shell scripting was not a good idea for production programs, I reviewed a smallish sample Bash script that had been deployed. I found a dozen latent bugs, 50% of which would have never have happened with Python (or Go, or...).

Re:What I learned from others' shell scripts by barrkel on Wednesday August 14, 2013 @12:57PM (#6212677)
Stringing together commands is the primary reason I write most of my scripts in bash; because my commands are generally written in multiple different languages. Also, the fact that the shell is a REPL, means scripts can be prototyped or even assembled from shell history fairly easily, then refactored into something better resembling usability.

Re performance, I usually run Cygwin on Windows. Starting up too many processes is not a mistake I tend to make, because forking in Cygwin is hopelessly slow. Similarly, spaces in paths are common, and I've learned to be fairly religious about quoting, to the point of using print0, xargs -0 etc.

My scripts regularly deal with millions of files, and pipes that transfer tens of gigabytes. I rely on being able to string together sort and uniq to do set operations over multi-gigabyte files with constant memory usage; such scripts are not trivially rewritten in languages like Python without using non-standard libraries. When performance becomes an issue, the solution is a lot more heavyweight in terms of development time.

Re:What I learned from others' shell scripts by unhammer on Wednesday August 14, 2013 @11:29AM (#6212086)
Fair enough; Unix certainly gives you enough ways to shoot yourself and your customers in the foot. I probably should've added the qualifier "wrt. succinctness" (that's e.g. the reason I move from awk to Python when I need more complex data structures than awk's arrays, which can often do the job, but not without looking ugly).

By the way, your "ls * .foo" does not even do the same as "find -name '*.foo'" due to a superfluous space ;-)

But the fact that most people don't know about xargs -P (or gnu parallel), or spawn too many processes, or use ugly hacks like pidfiles/start-stop-daemon, is not a reason to throw out all the good stuff shell scripting has to offer.

Re:What I learned from others' shell scripts by unhammer on Wednesday August 14, 2013 @11:36AM (#6212114)
By the way,

> "cat myfile | grep foo | awk '...'" […] indeed this is one of the most frequent performance sinks in shell scripts

Is it really? I would've thought loops were a more common performance sink. I can't imagine how that useless use of cat has _that_ much of an effect, unless you're running a whole bunch of copies of this script. It looks ugly in the process table and it does not let the real command (here: grep) move back and forth in the file, but I've never noticed performance improvements from removing uuoc's.

Re:What I learned from others' shell scripts by jzwinck on Friday August 16, 2013 @07:52AM (#6223387)
My original phrasing may have been unclear; with the elision you made the thrust becomes quite different. I intended to convey two thoughts: that Useless Use of Cat and other filters is pervasive because it is so easy, and secondly, that spawning tons of unnecessary processes is a common performance waste. The specific example with just three processes (cat, grep, awk) is not such a big deal--the bad cases are when stuff like that happens within loops (but the performance loss is not due to the loops, but rather the spawning).
Re:What I learned from others' shell scripts by e12e on Thursday August 15, 2013 @06:26AM (#6217008)
To be fair, there's (probably) two forks too many: awk can do filtering on regular expressions too.

Not that I think it really makes much of difference in most cases.

Re:What I learned from others' shell scripts by fizerkhan on Wednesday August 14, 2013 @03:36AM (#6210304)
Yes, you are correct. But some people crazy to write everything in shell scripts.
Re:What I learned from others' shell scripts by Fuxy on Wednesday August 14, 2013 @04:37AM (#6210472)
I find shell scripts confuses the heck out of me whenever they try to do something a little more complicated the piping.

Even this left me a but confused like how do the functions get the input parameters does the * in echo -e "$RED$*$NORMAL" have something to do with it?

There's just so many obscure ways of doing the same thing not to mention there's very few good places to teach you shell scripting and shell scripting best practices.

I'd rather just go with python. It's easier to read.

Re:What I learned from others' shell scripts by coolj on Wednesday August 14, 2013 @04:57AM (#6210509)
http://www.tldp.org/LDP/abs/html/

TLDP's Advanced Bash-Scripting Guide is a great resource for learning bash.

For example, this section talks about the meaning of "$*":

http://www.tldp.org/LDP/abs/html/internalvariables.html#ARGL...

Re:What I learned from others' shell scripts by barrkel on Wednesday August 14, 2013 @12:59PM (#6212693)
Free advice: you almost never want $. Almost always what you want is $@, usually in the form "$@". It doesn't matter for this example, but $ is usually a code smell.
Re:What I learned from others' shell scripts by unhammer on Wednesday August 14, 2013 @05:27AM (#6210584)
http://mywiki.wooledge.org/BashGuide is the ultimate bash guide, along with http://mywiki.wooledge.org/BashFAQ

For learning the differences between POSIX sh and bash (which can trip you up if you want to run on systems without bash, or if you simply don't know that #!/bin/sh is different from #!/bin/bash), see http://mywiki.wooledge.org/Bashism

And regarding how to parse your example, the $* in a string expands to all input parameters to the function, separated by the first character of the IFS variable (e.g. space).

Re:What I learned from others' shell scripts by unhammer on Wednesday August 14, 2013 @05:29AM (#6210588)
Also, there's no reason to use uppercase for all variables, it's just that environment variables are by convention uppercase so people seem to think their local vars need to be uppercase too.
Re:What I learned from others' shell scripts by 616c on Wednesday August 14, 2013 @04:52AM (#6210502)
Of course, there are also some people crazy enough to write web apps in C, C++, or dare I say Haskell! The latter has become popular, but every tool has a purpose, and every tool can be re-invented, like if you are a garbage social app so popularly social you compile your PHP into C++ and give it a music genre name.

I am sure no one will get that reference. :-)

Re:What I learned from others' shell scripts by cgag on Wednesday August 14, 2013 @05:04PM (#6214256)
What's crazy about using an extremely high level language (haskell) for web dev?
Re:What I learned from others' shell scripts by 616c on Thursday August 15, 2013 @01:14AM (#6216288)
It isn't as far as I am concerned, but it might upset some people.
Re:What I learned from others' shell scripts by kamaal on Wednesday August 14, 2013 @04:39AM (#6210478)
Python is an ideal replacement for small Java programs, not Shell.

The moment your script needs any more than 10 regular expressions, or dealing with >3 files all in a complex interplay- use of Perl becomes inevitable. And that is something like the very utmost basic thing you can do with Perl.

Python is more like tried-to-be-scripting-but-is-a-web-framework language.

Re:What I learned from others' shell scripts by icebraining on Wednesday August 14, 2013 @08:19AM (#6211054)
The moment your script needs any more than 10 regular expressions, you should probably take it out and shoot it.
Re:What I learned from others' shell scripts by rubinelli on Wednesday August 14, 2013 @07:32AM (#6210907)
I agree that Python is a general-purpose language that could replace much of the Java code you see around, but I never got the feeling that it is particularly well-suited for the web. For one thing, with significant whitespace, it's harder to embed snippets of code in HTML or vice versa, so you have to work with a templating library right from the beginning. Some will see this as a blessing in disguise, but for very small projects or mostly self-contained components, having logic and presentation in one file is much more productive.
Re:What I learned from others' shell scripts by kamaal on Wednesday August 14, 2013 @07:42AM (#6210939)
Ironically bulk of all the Python out there is web code.

In fact Django, Twisted and Zope is all the Python code there is.

Scripting was never Python's forte. Scripting is all about succinctness, power and providing as much power with fewer constructs and restrictions. Which happens to be exactly the very opposite of Python goals, and some thing which more or less as a mission statement Python tries to achieve. This is why Python will likely never be a very successful scripting language.

Python was always a web language for frustrated java programmers who couldn't put with java's problems anymore. Much of Python's success is in web frame work area. Which was previously Java's territory.

Re:What I learned from others' shell scripts by PommeDeTerre on Wednesday August 14, 2013 @08:23AM (#6211067)
Yes, Python is obviously used for web development. But it's very short-sighted, and even absurd, to claim that the "bulk of all the Python out there is web code".

It's even more absurd to claim that "Django, Twisted and Zope is all the Python code there is." That's utter nonsense, in fact.

There are numerous non-web applications that use Python extensively, whether they're partially or fully implemented in Python, or whether they can be scripted using Python.

Then there are the numerous libraries and frameworks for Python, from GUI toolkits through to scientific computation packages.

Many, many organizations use Python internally for a very wide variety of tasks and systems, without broadcasting such use loudly, if at all. This ranges from one-off scripts up to entire multi-million-line software systems.

I sure hope that you're joking, but it just isn't coming across as a joke. Nobody can seriously claim that "Python will likely never be a very successful scripting language" when it has undoubtedly been one of the most successful, and versatile, programming and scripting languages around for many years now.

Re:What I learned from others' shell scripts by kamaal on Wednesday August 14, 2013 @10:07AM (#6211547)
The biggest reason why Python 3 is/was seeing such late adoption is purely because web frameworks/libraries were not migrated to Python 3. Goes on to tell how many people use Python for what purposes, that absence of a domain of frameworks/libraries slows down the adoption of the language itself for years.

Of course Python is used for scripting purposes. But that amount of code is no where close the web code that is written in Python. It all depends how much code in ratio is written for what purposes and not the total amount of code written for that purposes.

Python's glory days came with web frameworks and continue to be the reason for its fame and wide spread adoption.

I will take Python seriously if it offers the same capabilities as Perl at least on the command line.

Python is a awesome general purpose language. But so far as scripting is concerned it is still no match to Perl

Re:What I learned from others' shell scripts by PommeDeTerre on Wednesday August 14, 2013 @06:10PM (#6214687)
Like I tried to explain to you earlier, those who use Django and the other Python web frameworks that may not really support Python 3 yet are a very small portion of the entire Python user community.

Many of the rest of us have been happily and very successfully using Python 3 for years now, including for the scripting tasks that you incorrectly claim we don't use Python for.

Your other claims like "Python's glory days came with web frameworks and continue to be the reason for its fame and wide spread adoption." are truly absurd and outright wrong. Python was very popular and widely used well before the mid-2000s, when many of the web frameworks you're referring to were first released.

I think you vastly overestimate the amount of Python used for web applications, as well. This is understandable, as such uses are often more visible than the other behind-the-scenes uses. But there's a staggering amount of Python code that you don't see, and that isn't used for web applications.

Re:What I learned from others' shell scripts by ctb_mg on Wednesday August 14, 2013 @12:18PM (#6212385)
> The biggest reason why Python 3 is/was seeing such late adoption is purely because web frameworks/libraries were not migrated to Python 3.

That is a pretty big assumption to make, in my opinion.

When we're talking in the context of scripting then we should be talking about what distros have easily accessible python3 packages/libraries, good python3 community support, and whether libraries relevant to sysadmin scripting tasks have been ported to python3 or not.

Re:What I learned from others' shell scripts by lotsofcows on Wednesday August 14, 2013 @09:22AM (#6211285)
Perl? I think you mean awk :-)
Re:What I learned from others' shell scripts by Millennium on Wednesday August 14, 2013 @10:23AM (#6211648)
Python is great for many things, but sometimes you need something lighter. That's not to say that Python is heavy -Java is heavy- but there are environments so constrained that even Python is not light enough. This is where shell scripts can come in handy.
Re:What I learned from others' shell scripts by michaelry on Wednesday August 14, 2013 @06:15AM (#6210696)
True if python is available, but on a small embedded system this is not always the case.