News for hackers. Stuff that matters.
Re:On env Shebangs by rgoulter on Friday April 15, 2022 @07:49PM (#31047396)
"Neither env shebangs nor absolute path shebangs are generally ‘better.’ Either one may get you in trouble when used in the wrong context."

I generally agree on the wisdom of "it depends and consider context", but..

An absolute path is going to assume others have the same setup.

The downside claimed for using #!/usr/bin/env seems to be:

- that the PATH needs to be setup correctly for interactive/non-interactive shells. (Which will be true of just executing that command in a script, anyway).

- and that the program might be a different version than expected (which an absolute path doesn't solve either).

Using #!/usr/bin/env is a much better default to go with, and will be the better option almost all the time.

Re:On env Shebangs by tyingq on Friday April 15, 2022 @07:57PM (#31047474)
One use case where #!/usr/bin/env can go wrong is with cron jobs. They don't typically pick up the same path as an interactive login or shell. So it's easy, for example, to get get a python 2 interpreter when you were expecting python 3 with "#!/usr/bin/env python"
Re:On env Shebangs by IshKebab on Saturday April 16, 2022 @05:05AM (#31050861)
You should always explicitly use `python3` rather than `python` and hoping it is version 3.
Re:On env Shebangs by tyingq on Saturday April 16, 2022 @09:49AM (#31052296)
Well, even then, you may not get the python3 you expect with "#!/usr/bin/env python3" in a cron job. I imagine a fair amount of this is happening now, with Xcode installing it's own python3 on a bunch of Macs.
Re:On env Shebangs by derefr on Saturday April 16, 2022 @12:26AM (#31049472)
> An absolute path is going to assume others have the same setup.

Certain absolute path locations are standardized by POSIX. Including the absolute path of /usr/bin/env.

> and that the program might be a different version than expected (which an absolute path doesn't solve either).

...and those programs in those standard locations are required by those same standards to behave ABI-compatibly with their behavior at time of standardization.

Which is why e.g. /bin/bash bothers to strip out features when called as /bin/sh — the interpreter at /bin/sh must be no less and no more than the particular feature-set of the Bourne shell version that POSIX was standardized upon.

Re:On env Shebangs by sp1rit on Friday April 15, 2022 @07:37PM (#31047291)
A cool thing rpm(build) does, is that is automatically detects shebangs of packaged files and makes the built package depend on whatever package provides the file the shebang points to. This obviously cannot work for env shebangs, so you always have to do that find -exec sed dance to replace them with proper ones for software that decides to have env shebangs.

But on Linux we also don't have to fight with a 15 year old bash or "compromising" a system when installing something into /bin. It feels lile OS X genuinely doesn't sound like a good UNIX to administer.

Re:On env Shebangs by Filligree on Friday April 15, 2022 @08:10PM (#31047574)
And what of other dependencies? Imagemagick, say? There's another way to achieve portability and exactness, although it requires some setup.

  #!/usr/bin/env nix-shell
  #!nix-shell -i bash -p imagemagick
Voila. A (mostly) self-contained shell script, dependencies auto-installed when needed.
Re:On env Shebangs by tuatoru on Friday April 15, 2022 @11:38PM (#31049211)
> another way to achieve portability

    $ nix-shell
    nix-shell: command not found
Re:On env Shebangs by abathur on Friday April 15, 2022 @11:19PM (#31049096)
I came here to say this, too :)

But, of course, it isn't a silver bullet...

1. You still have to have a sane PATH. A fair amount of the Nix install-related issues that get opened are PATH problems, and you can also run into problems with PATH in cron/launchd (as @tyingq noted).

2. You still have to know what the script depends on. This can get tricky beyond small scripts you wrote yourself. (I write a tool for discovering and ~linking/resolving external dependencies in Shell scripts, https://github.com/abathur/resholve. As I've been working on converting some of nixpkgs' existing Shell packages to use it, I almost always find dependencies the initial packager missed.)

Re:On env Shebangs by Filligree on Saturday April 16, 2022 @04:05AM (#31050563)
Re. 2-

If you invoke nix-shell with --pure, then it'll erase most user state — PATH included — prior to invoking the script. I don't find much use for this, but it would be useful to ensure nothing that isn't an explicit dependency gets used.

Re:On env Shebangs by abathur on Saturday April 16, 2022 @08:56AM (#31051934)
Yes, `--pure` helps (and `env -i` can also help) but that's not the hard part of finding them all. It only really helps when there's no conditional logic or no options that can lead to the execution of different programs.
Re:On env Shebangs by watersb on Friday April 15, 2022 @11:02PM (#31048971)
Interesting.

I've resisted the pull of the Nix side so far...

There's a similar package that auto-pulls dependencies for Swift scripts, but it requires you to explicitly state the external dependencies as comment annotations. The example given is:

    #!/usr/bin/swift sh
    
    import Foundation
    import PromiseKit  // @mxcl ~> 6.5
    
    firstly {
        after(.seconds(2))
    }.then {
        after(.milliseconds(500))
    }.done {
        print("notice: two and a half seconds elapsed")
        exit(0)
    }
    
    RunLoop.main.run()


https://github.com/mxcl/swift-sh
Re:On env Shebangs by kazinator on Friday April 15, 2022 @07:42PM (#31047333)
I'd never use /usr/bin/env to find a shell. Just #!/bin/sh and write portable code, end of story.

The env trick is useful when you have some programming language that is not commonly packaged. Users might have it in /usr/local/bin or some completely personal place that's in their PATH: /home/joe/bin or what have you.

Re:On env Shebangs by davvid on Saturday April 16, 2022 @03:14AM (#31050306)
The git project uses shell extensively and the resulting tools are portable to just about every unix under the sun and the Git for Windows bash environment.

Their shell script guidelines are pretty helpful for someone wanting to familiarize themselves with portable posix shell scripting. There's some style nits in there too, but the other stuff can be helpful if you're looking for a well-traveled path.

https://github.com/git/git/blob/v2.35.0/Documentation/Coding...

Re:On env Shebangs by remram on Saturday April 16, 2022 @09:22AM (#31052110)
The overwhelming majority of scripts with such a shebang only works on bash with GNU tools. Trying to run them on dash or with BSD tools is often impossible. When it is compatible, it often has critical bugs due to the limitations of error handling (pipefail and $PIPESTATUS are both nash extensions).

Please do write Python/Perl instead.

Re:On env Shebangs by kazinator on Sunday April 17, 2022 @12:48AM (#31058586)
I can easily write a shell script by following the 1997 Single Unix Specification Issue 2:

https://pubs.opengroup.org/onlinepubs/007908799/

put it into a #!/bin/sh file and have it work everywhere.

Good luck with #!/usr/bin/env python followed by code written to 1997 specs.

Re:On env Shebangs by remram on Sunday April 17, 2022 @05:53PM (#31064831)
When people say "portable" they don't usually have 1997 computers in mind. Following the old spec is good to have broad support on today's machines but is not in itself a target.

A script written for Python 3.5 (which was released 7 years ago) is very portable and will even work on non-Unix platforms like Windows. If it's not there, it is easier to compile than the entire suite of Unix tools. In addition, it has proper error handling...

Re:On env Shebangs by nerdponx on Monday April 18, 2022 @11:02AM (#31071595)
The nice thing about Python is that, for the most part (see below for exceptions), old 3.x code is forward-compatible to new code. So a basic Python 3 script written in 2008 should still run fine in 2022.

Exceptions:

Asyncio has been in active development and was considered "provisional" for a long time, but is now stabilizing. Asyncio code that predates async/await syntax will not run on new versions of Python (this maybe can be converted with automated tools).

Type annotations have also been in active development and was considered "provisional" for a long time, and is also now stabilizing. In a future version (maybe 3.11?) a lot of `from typing import ___` imports will break and need to be replaced (this can mostly be converted with automated tools).

Several modules in the standard library are considered "dead batteries" and will be removed in an upcoming version. See <https://peps.python.org/pep-0594/>, and for discussion <https://discuss.python.org/t/pep-594-take-2-removing-dead-ba...>. This is probably the worst aspect for forward-compatibility, but it seems necessary to ensure the health of the CPython project going forward (unless of course enterprises decide to start contributing back some faction of the billions upon billions that they make on the backs of the Python Software Foundation every year). Expect community members to spin off these libraries

Re:On env Shebangs by nerdponx on Monday April 18, 2022 @11:30AM (#31072035)
It's also worth mentioning that there are "higher-level" languages than C, which produce portable-ish native binaries (modulo some dynamic linking dependencies, which isn't that different from depending on a Python or Perl runtime).
Re:On env Shebangs by 0xbadcafebee on Friday April 15, 2022 @07:54PM (#31047444)
In some cases the env shebang is basically mandatory. If you use virtualenv in a Jenkins job's workspace with a long job name, you can end up with shebangs that are longer than the kernel can execute (about 127 characters on most systems[1]). The solution is to rewrite the virtualenv-created scripts to use the env shebang, and set the PATH to the virtualenv bin directory before executing the scripts.

[1] https://stackoverflow.com/questions/10813538/shebang-line-li...

Re:On env Shebangs by cryptonector on Friday April 15, 2022 @07:41PM (#31047321)
Sadly you can only portably have one argument in the shebang line to the interpreter. That means that if you wanted to pass an argument to the actual interpreter, you'd not be able to if you use the env trick.
Re:On env Shebangs by lucb1e on Friday April 15, 2022 @07:57PM (#31047469)
Is this not what you are looking for? From the man page of env(1)

> The -S option allows specifing [sic] multiple parameters in a script.

> [...]

> Without the '-S' parameter the script will likely fail with:

> . . . /usr/bin/env: 'perl -w -T': No such file or directory

I actually ran into this problem last week and just used the absolute path with an argument (/usr/bin/python3 -u) instead of passing an argument to env, and I added a TODO comment to figure out how to do this better. Now you made me find the right way :)

Re:On env Shebangs by sibrahim on Friday April 15, 2022 @08:03PM (#31047529)
The usual next difficulty is that the maximum shebang length is fairly low (truncated at 128 chars, IIRC)
Re:On env Shebangs by lucb1e on Friday April 15, 2022 @08:16PM (#31047622)
It was funny to notice my mental knee-jerk reaction of "ought to be enough for anyone" (referring to 640K RAM). I've never used args with env until last week in the first place and even then it was a (mental) debate of solving it in code (wrapping print() in a function that also flushes stdout) or adding the argument instead, so I guess that's why that was my initial thought. But of course that doesn't mean there can't be legitimate uses of long argument lists. Java code comes to mind (although I do question the sanity of many things Java).
Re:On env Shebangs by cryptonector on Friday April 15, 2022 @07:59PM (#31047499)
Ahh, thanks!
Re:On env Shebangs by kazinator on Friday April 15, 2022 @08:01PM (#31047508)
This is the fault of interpreters. Interpreters receive the entire file as input, with the shebang line and all. That can easily support the encoding of additional parameters.

In TXR Lisp, I invented a novel way of doing this, which I call the "hash bang null hack".

If you put a NUL character into the hash bang line, then TXR will process arguments after that character. The NUL character is a string terminator, and so the operating system kernel doesn't see it; it thinks that's the end of the hash bang line.

So for instance (^@ represents NUL):

#!/usr/bin/env txr^@arg arg arg

This hack is not necessary, of course; a given language interpreter can do this in any number of ways. E.g. arguments in a comment in the following line. The convention could be, say, that if the next line also starts with #!, it holds parameters:

  #!/usr/bin/env my-language
  #!--args to my language
I didn't do it this way because the language doesn't use # comments (so hacking argument recognition into comment processing is out) and I didn't want to complicate the hash bang logic to dealing with two or more lines: everything is handled in the one line and that's it.

Another solution is to rely on env having the BSD -S option, which specifies a string that becomes multiple arguments.

  #!/usr/bin/env -S program --option arg
the hash bang mechanism will pass "program --option arg" as a single argument to env, whose -S option will split that into multiple arguments. You might run into a limit on the hash bang header. E.g. if the given operating system reads only, say, 32 bytses of the file (really ancient Unix system), in which it looks for the hash bang line, then you have only that much space to cram into that -S argument.

The -S option implemented in BSD unixes has a lot of bells and whistles: you can use escape characters in it to encode control characters and $ to expand environment variables. E.g. silly example:

   #!/usr/bin/env -S $USER/bin/interp
would use every user's own "interp" in their own bin directory to run the program.

-S is not portable; it's not specified in POSIX. Newer versions of GNU Coreutils have it (but not with all the BSD features, I think). If you maintain a programing language, it behooves you to solve this problem for the users, so the only nonstandard thing they rely on is your language, without any collateral nonstandard stuff.

Re:On env Shebangs by LeoPanthera on Friday April 15, 2022 @07:50PM (#31047406)
Assuming bash, you can specify arguments using "set". For example:

  #!/usr/bin/env bash
  set -eu
  
  # Your script here
Re:On env Shebangs by KSPAtlas on Friday April 15, 2022 @09:05PM (#31048019)
/usr/bin/env is also the only shebang you can really use on nixOS AFAIK
Re:On env Shebangs by assbuttbuttass on Friday April 15, 2022 @09:36PM (#31048276)
#!/bin/sh is also available
Re:On env Shebangs by jbverschoor on Saturday April 16, 2022 @03:28AM (#31050392)
> The /usr/bin/env shebang provides a means for system portability.

No it doesn’t. It’s just there to specify which interpreter to use instead of the current shell, so you can rub it like any other executable. Has nothing to do with portability.

Re:On env Shebangs by _moof on Saturday April 16, 2022 @03:31AM (#31050404)
You're talking about shebangs in general. This is about a specific shebang idiom, i.e

  #!/usr/bin/env python3
which allows the file to be run with whichever python interpreter the user has in their PATH, instead of requiring the interpreter to be in a particular location.
Re:On env Shebangs by jbverschoor on Saturday April 16, 2022 @06:05AM (#31051119)
Oh yeah you're right.. Didn't actually notice env xxx