News for hackers. Stuff that matters.
Re:Bash $* and $@ (2017) by koala_man on Sunday January 12, 2020 @02:50PM (#22028736)
I keep posting this, but my favorite rule of thumb came from a Google dev infra engineer who said that "every Python script over 100 lines should be rewritten in bash, because at least then you won't fool yourself into thinking it's production quality"
Re:Bash $* and $@ (2017) by yodsanklai on Sunday January 12, 2020 @03:51PM (#22029225)
The Google shell style guide [1] says this:

"If you are writing a script that is more than 100 lines long, you should probably be writing it in Python instead. Bear in mind that scripts grow. Rewrite your script in another language early to avoid a time-consuming rewrite at a later date."

[1] https://google.github.io/styleguide/shell.xml

Re:Bash $* and $@ (2017) by cookiecaper on Sunday January 12, 2020 @08:29PM (#22031127)
Stuff like this does a huge amount of unintentional damage. Every crappy company in the world thinks they're Google and tries to copy them. Google-caliber people might be able to be reasonable about this, but ordinary employees come across something like this and interpret it to mean "bash is evil" and ostracize everyone who tries to write a bash script for anything, no matter how sensible.

At work lately, there's been a spate of contorted "just why?" Python scripts that could've been accomplished elegantly in a handful of lines of shell. While no one would select shell languages as the ideal for a lot of complex logic, data parsing, etc., there's no competition for a shell when you need to do what shells are meant to do: chaining invocations, gluing and piping output, and so on.

Re:Bash $* and $@ (2017) by Godel_unicode on Sunday January 12, 2020 @09:56PM (#22031476)
Just because they write crappy Python doesn't mean they'd write good shell scripts. For me it indicates the opposite.
Re:Bash $* and $@ (2017) by samtheprogram on Monday January 13, 2020 @12:59AM (#22032179)
As Gary Bernhardt said, and I quote often, it makes sense when it’s right half of the ass.

https://youtube.com/watch?v=ZQnyApKysg4

Re:Bash $* and $@ (2017) by pmarreck on Tuesday January 21, 2020 @02:15PM (#22110350)
wow, have seen many of Gary's talks but not this one until now; clearly I missed a gem
Re:Bash $* and $@ (2017) by pif on Monday January 13, 2020 @04:11AM (#22032970)
> Google-caliber people might be able to be reasonable about this...

I'm not sure about that. Google certainly has some very talented people among their developers, but most of them are so bad at programming that they had to invent their own heavily watered down programming language.

Re:Bash $* and $@ (2017) by pjc50 on Monday January 13, 2020 @05:56AM (#22033387)
Shell is brilliant until you need to keep a filename in a variable. Or, worse, read filenames from the output of a command. Since basically everything except '/' and '\0' is a valid filename character, the potential for mishaps or exploits is huge.

That's where we came in with the need to use "$@" vs the unsafe-by-default bare $@.

Re:Bash $* and $@ (2017) by TazeTSchnitzel on Monday January 13, 2020 @01:07PM (#22036929)
> here's no competition for a shell when you need to do what shells are meant to do: chaining invocations, gluing and piping output, and so on

There is no reason this shouldn't be easy to do in Python too, if it is then it is something a library should fix.

Re:Bash $* and $@ (2017) by DonHopkins on Monday January 13, 2020 @01:23AM (#22032270)
But bash IS evil.
Re:Bash $* and $@ (2017) by cookiecaper on Monday January 13, 2020 @07:34PM (#22040488)
I mostly agree, but it's sometimes a necessary evil. There's a reason I praised shell languages in general rather than bash specifically; fish shell, for instance, is much more comfortable.
Re:Bash $* and $@ (2017) by cpach on Monday January 13, 2020 @05:55AM (#22033383)
Why?
Re:Bash $* and $@ (2017) by pmarreck on Tuesday January 21, 2020 @02:17PM (#22110375)
In most languages, you don't need to know/understand this much minutiae about string handling; as an example, I've been doing open source dev for years now in Bash and I never knew the distinction OP posted.
Re:Bash $* and $@ (2017) by a3n on Sunday January 12, 2020 @04:15PM (#22029430)
Related, the longer your shell script gets, the greater potential for accumulated foot-guns. When you finally do the rewrite, will you be bug-compatible? And once discovered, will you be confident on which side the bug lies?
Re:Bash $* and $@ (2017) by blaser-waffle on Monday January 13, 2020 @09:03AM (#22034511)
An aside: when did the phrase "foot-guns" become a thing.

Cuz I've never heard it until this thread, where it pops up multiple times. Great phrase but went 0-60 in usage.

Re:Bash $* and $@ (2017) by mrguyorama on Monday January 13, 2020 @11:39AM (#22035951)
I'm pretty sure it's at least as old as usenet and mail lists
Re:Bash $* and $@ (2017) by HocusLocus on Sunday January 12, 2020 @04:19PM (#22029475)
"If you are writing a script that is more than 100 lines long, you should probably be writing it in Perl instead."

This is true, especially of Python scripts.

Re:Bash $* and $@ (2017) by RHSeeger on Sunday January 12, 2020 @06:09PM (#22030357)
That way, it will never be rewritten again; because nobody will be able to read it.

Edit: Getting some downvotes, so to clarify, I am indeed saying that concerting something from Python to Perl has a high likelihood of making it for less maintainable. I get that people can write good perl. As someone who has had to maintain perl in the past; the fact is that it's far more common for the end result to be horrible perl. I have some issues with Python, but it is FAR more maintainable than perl.

Re:Bash $* and $@ (2017) by bsder on Sunday January 12, 2020 @07:21PM (#22030796)
Let me drag this to something concrete: if you are interviewing, it is almost impossible to pass an amateur "Perl screen".

Every single person who doesn't use Perl every day, professionally knows a different subset of Perl. This is one of the few times where being more knowledgeable than the interviewer is also a problem. You will write something, and the interviewer will question you on it because he has never seen it.

I finally solved this problem by bringing one of my personal programs written in Perl and also rewritten in Python so we could talk about it. Now, the interviewer is in MY subset of Perl AND can't argue because I have working Perl code in front of him. This was back in the 1990's when everybody in VLSI design expected you to know Perl.

Because of this "different subset" issue, if you want to maintain a Perl script, you have to basically know the entire language. This is what makes maintaining Perl scripts so difficult.

This "different subset" problem is the whole reason I left the Perl ecosystem back in 1996(!) at the height of Perl's popularity and never looked back.

Re:Bash $* and $@ (2017) by asveikau on Sunday January 12, 2020 @08:18PM (#22031076)
> This is one of the few times where being more knowledgeable than the interviewer is also a problem

I have actually found that is pretty much always a problem. If you pull out something the interviewer is unfamiliar with they will often assume you are full of it. I have had such people refuse perfectly good explanations for things they haven't heard of because they assume incompetence before that possibility.

Re:Bash $* and $@ (2017) by laumars on Monday January 13, 2020 @03:07AM (#22032695)
That's a sign of a bad interviewer and thus also an indicator of potential bad management.

I'd take those experiences as a blessing rather than a curse.

Re:Bash $* and $@ (2017) by cookiecaper on Monday January 13, 2020 @04:06PM (#22038584)
Indeed. I've "failed" interview screens because the reviewer's Python installation was broken, and rather than reading the traceback and realizing this, they just assumed the candidate's code was broken.

Last I knew, that position was still open almost a year after the fact.

Re:Bash $* and $@ (2017) by cosmojg on Sunday January 12, 2020 @07:57PM (#22030986)
As an experienced Perl programmer, what's your opinion of Raku? Is it a worthwhile replacement or too little too late?
Re:Bash $* and $@ (2017) by redis_mlc on Sunday January 12, 2020 @08:29PM (#22031122)
> As an experienced Perl programmer, what's your opinion of Raku?

Perl5 still works fine, and is maintained.

> Is it a worthwhile replacement or too little too late?

False dichotomy - see above.

Re:Bash $* and $@ (2017) by DonHopkins on Monday January 13, 2020 @01:29AM (#22032302)
It's been too little too late for many years, while it was still named Perl 5 for so long.
Re:Bash $* and $@ (2017) by frank2 on Tuesday January 14, 2020 @04:23AM (#22043168)
Raku is the new name for Perl 6, not Perl 5.
Re:Bash $* and $@ (2017) by arbie on Sunday January 12, 2020 @10:09PM (#22031527)
> back in the 1990's ... everybody in VLSI design expected you to know Perl.

Has this changed at all?

Re:Bash $* and $@ (2017) by bsder on Monday January 13, 2020 @02:00AM (#22032428)
Dunno. I finally left VLSI design because it was effectively a career dead end. Sense a trend? :)

While I still regard myself as a vastly better VLSI designer than programmer, my ability to wrangle software the whole way from assembly language on a chip to just shy of the top of a full web stack pays far better than my ability to wrangle transistors. And, in my opinion, attacks far more interesting problems.

Re:Bash $* and $@ (2017) by echelon on Sunday January 12, 2020 @03:31PM (#22029092)
That's a cute heuristic, but I think the better practice is to distrust scripts without tests as they can quickly diverge from the rest of the codebase.

It's still better to script in Python or Ruby than Bash. Nobody understands Bash. It's even more mysterious than Perl.

Re:Bash $* and $@ (2017) by fiddlerwoaroof on Sunday January 12, 2020 @04:30PM (#22029578)
I’d rather write bash for orchestration than nearly anything else: bash is designed to make coordinating processes easy, something very few programming languages have managed to do.
Re:Bash $* and $@ (2017) by laumars on Sunday January 12, 2020 @04:55PM (#22029792)
The thing that gets me about all the new shells and shell scripting languages popping up these days is they loosely seem to fall into 2 categories:

1. more emphasis traditional programming paradigms (be it JS, LISP, Python, whatever) which leaves a platform that is arguably a better designed language but however is a poorer REPL environment for it. Bash works because it's terse and terseness is actually preferable for "write many, read once" style environments like an interactive command prompt.

2. or they spend so much effort supporting POSIX/Bash -- including their warts -- that they end up poorer scripting languages.

I think what we really need isn't to rewrite all our shell scripts in Python but rather better shells. Ones which work with existing muscle memory but isn't afraid to break compatibility for the sake of eliminating a few footguns. Shells that can straddle both the aforementioned objectives without sacrificing the other. But there doesn't seem to be many people trying this (I can only think of a couple off hand).

Re:Bash $* and $@ (2017) by lukeschlather on Sunday January 12, 2020 @08:22PM (#22031092)
I've been writing a lot of Powershell lately, and my only real gripe with it is that it seems suspiciously like not being Posix compliant in any way was a design goal.

I agree with the idea of breaking backwards compatibility, but Powershell honestly has enough core design issues that it itself is starting to feel like it needs a major backwards-incompatible update.

Re:Bash $* and $@ (2017) by leosarev on Sunday January 12, 2020 @11:23PM (#22031804)
Major backwards-incompatible update have come recently (Powershell Core).
Re:Bash $* and $@ (2017) by nailer on Monday January 13, 2020 @06:17AM (#22033468)
What did they break? Powershell core just doesn't include Win32, but that's fine, it runs on Linux and macOS and they don't need Win32.

Powershell 7 has a full Win32.

Re:Bash $* and $@ (2017) by leosarev on Saturday January 18, 2020 @03:54PM (#22087125)
I.e. curl command now means actual curl, not Invoke-webrequest
Re:Bash $* and $@ (2017) by fiddlerwoaroof on Sunday January 12, 2020 @04:58PM (#22029811)
If you haven’t tried scripting in zsh, I’d give it a go. zsh has a lot of safety improvements and quality of life features.

https://news.ycombinator.com/item?id=22029662

Re:Bash $* and $@ (2017) by laumars on Sunday January 12, 2020 @05:13PM (#22029910)
It's also subject to many of the same footguns as Bash so I'd put that into the 2nd camp (re my previous post).

Not that I'm taking anything away from zsh. It is a nice shell. But I think we can do even better considering how dependant we still are on shells for day to day stuff.

Re:Bash $* and $@ (2017) by fiddlerwoaroof on Sunday January 12, 2020 @05:22PM (#22029978)
> Zsh had arrays from the start, and its author opted for a saner language design at the expense of backward compatibility. In zsh (under the default expansion rules) $var does not perfom word splitting; if you want to store a list of words in a variable, you are meant to use an array; and if you really want word splitting, you can write $=var.

https://unix.stackexchange.com/a/26672

Re:Bash $* and $@ (2017) by fiddlerwoaroof on Sunday January 12, 2020 @05:21PM (#22029969)
I realize that it has similar footguns, however reading through their info pages, I was surprised by how many they just decided to fix, unless you explicitly turn on compatibility mode.
Re:Bash $* and $@ (2017) by jhasse on Sunday January 12, 2020 @04:57PM (#22029804)
What are your thoughts about fish?
Re:Bash $* and $@ (2017) by fiddlerwoaroof on Sunday January 12, 2020 @04:59PM (#22029820)
Does fish still have the issue where pipelines aren’t really concurrent?
Re:Bash $* and $@ (2017) by XorNot on Monday January 13, 2020 @07:54AM (#22033993)
Go is actually superb at this IMO. Channels and goroutines provide the actual primitives you need to handle streaming data, and the integration of the context library makes starting up and shutting down everything a breeze.

Best of all though, it's absolutely compatible wherever you need to run it.

Re:Bash $* and $@ (2017) by fiddlerwoaroof on Monday January 13, 2020 @05:49PM (#22039618)
The issue with I’ve had with go is that it’s batch compiled: I can repl something together in an interactive shell and then generalize it, which is my preferred workflow
Re:Bash $* and $@ (2017) by spc476 on Sunday January 12, 2020 @09:39PM (#22031409)
Here's one thing that I find difficult to remember how to do in shell: run a command, have stdout go to a file (or /dev/null or whatever), but pipe stderr to more or grep or some other program to process.

I mean, I would expect the syntax to be something like:

    make >/dev/null 2| more
but no ... it's some incomprehensible mess of redirection arcana to get that to work.
Re:Bash $* and $@ (2017) by pwg on Sunday January 12, 2020 @10:07PM (#22031522)
The syntax for that is present in the manual:

    Note that the order of redirections
    is significant.  For example, the 
    command

    ls > dirlist 2>&1

    directs both standard output and 
    standard error to the file dirlist, 
    while the command

    ls 2>&1 > dirlist

    directs only the standard output to 
    file dirlist, because the standard
    error was duplicated from the 
    standard output before the standard 
    output was redirected to dirlist.
Does it look a little arcane, yes, esp. until one memorizes it.

Can it be memorized? Yes, because it is just a single 'incantation': "2>&1". Just put that redirection operator before the redirection of the standard output to the file, and the result is stdout goes to the file, stderr goes to the pipe.

Re:Bash $* and $@ (2017) by Legogris on Monday January 13, 2020 @03:33AM (#22032799)
Just in case someone's confused by this, it becomes clear if you know how pipes work. `2>&1` means "redirect stderr to whatever stdout is currently pointed to" (redirection by copy, not by reference). More recent version of bash manual[0] has a less confusing wording in the same paragraph IMO:

  Note that the order of redirections is significant. For example, the command

  ls > dirlist 2>&1

  directs both standard output (file descriptor 1) and standard error (file descriptor 2) to the file dirlist, while the command

  ls 2>&1 > dirlist

  directs only the standard output to file dirlist, because the standard error was made a copy of the standard output before the standard output was redirected to dirlist. 

[0]: https://www.gnu.org/software/bash/manual/html_node/Redirecti...
Re:Bash $* and $@ (2017) by dllthomas on Sunday January 12, 2020 @11:14PM (#22031759)
It's actually a pretty simple model. Every process has an array of open file objects. File descriptors are indexes into that array. "Redirection" copies the underlying entries, and are processed left-to-right.

"Redirecting streams" winds up being confusing - it's all just `dup2`.

Re:Bash $* and $@ (2017) by spc476 on Monday January 13, 2020 @05:17AM (#22033249)
Okay, but can you pipe stdout to one string of commands, and stderr to a different string of commands? That's something I feel should be possible but how Unix shells handle redirection is just ... alien to my way of thinking.

Yes, I know that underneath it's all calls to `pipe()` and `dup2()`, which I can do (and have done) in a language other than shell. It's the shell redirection syntax (for anything more complex than simple redirection or a pipe) that just doesn't make sense to me.

Re:Bash $* and $@ (2017) by pwg on Monday January 13, 2020 @07:19AM (#22033773)
Yes you can (provided you have Bash and a system that supports process substitution):

    command 2>&1 > >(cat - | cat - > fd1) | cat - | cat - > fd2
Note, the extra 'cats' are just to show a "string of commands".

How to read this:

When Bash sets up the file descriptors for 'command' it initially makes descriptor 1 refer to the pipe, and descriptor 2 refer to the terminal.

So, the dup operator (2>&1) copies the fd in descriptor 1 (which is the pipe) into descriptor 2 (so after the dup operator is processed both stderr and stdout for command reference the pipe).

Next descriptor 1 is replaced (the > operator) by a reference to a fifo created by the process substitution operator (the >(...) operator).

So now command's descriptor 1 refers to the fifo created by >() (which itself contains a "string of commands").

Then, because stderr for command was made to be a copy of what was previously stdout (before modifying stdout) it continues to refer to the pipe Bash setup (the | operator), and stderr now flows out over the pipes to another "string of commands".

Re:Bash $* and $@ (2017) by dredmorbius on Sunday January 12, 2020 @11:26PM (#22031818)

    ( echo "stdout"; echo "stderr" 1>&2 ) 2>&1 1>/dev/null
Re:Bash $* and $@ (2017) by echelon on Monday January 13, 2020 @12:47AM (#22032134)
Practically alien runes.
Re:Bash $* and $@ (2017) by reagent_finder on Monday January 13, 2020 @06:38AM (#22033547)
I want to disagree with this so badly because I have tried to do stuff that's just a bit too complex for bash and ended up scrapping the idea... but I can't. Hacky python is even worse.