Or at least, they shouldn't be putting it in the $PATH.

Whether you're using Linux, macOS, or BSD, OS-packaged language runtimes exist for the OS itself to use, in OS-internal scripts and underpinning OS services. They shouldn't be considered "part of" the userland; as this invites userland applications to rely on them. These runtimes are actually something more like an "implementation detail" of the OS — the kind of thing whose binaries should live in $PREFIX/libexec rather than $PREFIX/bin.

(Mind you, if you're a system software developer, whose software gets packaged into Operating Systems, then you do have to rely on this "implementation detail." Which often means targeting very old versions of runtimes and libraries, compared to the versions that application developers get to target.)

Also, of course, there are "standard" language runtimes that should be part of the userland — e.g. /bin/sh. But these are literally standard — i.e. part of some standard like POSIX, that an OS can be tested for conformance against. Which is important, because it means that developers can write to a stable target of "POSIX compatible", assuming the existence of (and particular stable semantics of) particular standard runtimes.

But none of the "modern" language runtimes are part of any such standard. The version OSes ship should be assumed to not exist. (And major third-party applications do already do the right thing, in not making any assumptions of existence, but rather installing their own vendored versions of the runtimes that they can upgrade as needed.)

I’ve always thought it strange that most Linux distributions and other *nixes don’t put up barriers between not only system and userland, but also between “essential” userland (i.e. DEs) and other user-installed packages. The lack of separation seems like a setup for failure and complicates package manager design considerably.

This is where the excitements of using linux for day to day tasks come from, right? Install Steam and suddenly your DE got uninstalled [1], for example.

[1] https://www.reddit.com/r/linuxmemes/comments/qhjhf2/linus_tr...

But it really isn't. I never have problems in linux with python apps that are prepackaged by the distro. It's up to you to keep that separate if you install your own apps that require python. You really can't blame it on distro maintainers as they are really doing just fine with it. What happens is people won't take the time to understand what they're running instead of randomly typing a CLI or clicking on the app icon. Linux doesn't hold your hand (and really it can't! there are simply too many possible combinations) unless you use what comes with the distro.

> unless you use what comes with the distro.

And, even then, it's not the best choice for users that need hand-holding.

Can you come up with a solid definition to separate "essential" userland from other user-installed packages?

You cite DEs as essential, but maybe I want to be using my own Enlightenment install rather than whatever the system has installed (true story!)

Well I mean, there’s no reason why the user shouldn’t be able to control what’s installed as essential userland. It just needs to be managed entirely separately from the system and “normal” userland. It could be as simple as a flag on apt, dnf, etc.

This is the reason I really, really love Nix. I have been fighting this same battle for years and Nix finally fixed it.

I'd define it as "never have zero packages that provide a DE". This could be a global setting in the package manager which you would be able to set to `false` if so desired.

You can always build escape hatches for that kind of use case. Some more stringent separation between essential and installed components could have prevented Linus removing his DE.

What's a DE?

Short for Desktop Environment. You can choose yours in Linux.

By the same token, they should remove bash and libc from $PATH as well, as these are also runtimes that the OS itself uses. I think for a long time Python was just as much part of Ubuntu's intended user facing API as those two or GNU ls. Until the Python 3 came and split the Python community in two, there were few reasons to complain. If you want to package your own Python with your app, nothing stops you from doing so and calling it explicitly.

> I think for a long time Python was just as much part of Ubuntu's intended user facing API as those two or GNU ls.

To be clear, it doesn't really matter what Ubuntu's user-facing API is, because nobody writes a software package for Ubuntu specifically. They write it for, at least, Linux; or more likely, the de-facto subset of POSIX that includes at least Linux, macOS, and Windows MinGW. And that "Linux" target, almost always implies intended portability to "weird" distributions like CoreOS or Alpine Linux.

When you create a portable upstream software package that doesn't make any assumptions about where it's going to be installed, you can still rely on both the existence and the (POSIX subset) semantics of ls! But Python? No way.

In fact, you can't actually rely on libc being any particular way, either — but that's what autoconf is for. (Which in turn relies on the existence of, and standard POSIX semantics for, particular tools like m4.) Given that you have those POSIX-standard tools, you can generate a configure script. But you don't even need those tools to run the configure script. It just relies on, IIRC, POSIX-standard compatibility-mode cc(1). That's everything the configure script needs to probe the behavior of your system's libc (and kernel library headers, and a bunch of other things.) As such, a configure script, once generated by autoconf on some system somewhere, is portable to literally any POSIX with a compiler (which is an incredible feat requiring some horribly ugly hacks.)

As for bash: it's not something you can assume. When writing portable code, you can assume (re: POSIX) that whatever shell is at /bin/sh is a compatible implementation of the Bourne Shell; but you cannot assume that that shell is specifically the Bourne Again Shell, or that it can parse Bourne Again Shell syntax/semantics. Under Busybox, for example, /bin/sh is a compatibility mode of https://en.wikipedia.org/wiki/Almquist_shell, and bash-isms don't work. (Ever written a script to run on a Synology NAS? Ash is all you get.)

The moment you start talking about C compilers you've stopped talking about most software running today on Linux systems. The vast majority of packages are delivered as apt/rpm/apk etc installers, which are overwhelmingly binary or interpreted code. If you're delivering an apt package, you will want to assume (or ensure through declared dependencies) some things about the target system, such as bash or Python.

And you'll find numerous bash-only scripts online, either as examples, helpers, stack overflow answers, and many others. Comparatively few bother to support POSIX sh - though that trend is somewhat reversing with the rise of containers and Alpine Linux and BusyBox with them, plus Apple's moves against bash.

Just like you will find no tutorials telling you to edit a file with ed, even though that is the standard text editor.

Finally, if you think it's in any way common to write software to support any POSIX-compatible system, or even any Linux system, you live in a much nicer corner of the world than I do.

Not the same token, bash and libc are backwards compatible.

I already acknowledged the misguided Python 3 split made this into a problem. But that is not the reasoning offered by the GP - they were claiming that Python was only an implementation detail of the OS and shouldn't have been exposed in the first place, which makes no sense to me.

"They shouldn't be considered "part of" the userland; as this invites userland applications to rely on them."

Or should they? Windows applications rely on .net runtime that is part of the OS. For years now. It's one of the reasons why application development on Windows can be so laid back and stable.

I really, really don't understand why people in Unix-land feel the need to move fast and break things.

My understanding is that once they remove python2, there won't be any Python bundled. There will be a python3 handler that just prints a message asking you to use `xcode-select` to install the XCode command line tools, and that'll include the actual python3... but that's a fairly up-to-date version these days. (Right now it's 3.8.9.)

I fully agree with removing Python from OSes, but I don't know that it's reasonable for the average user to manage their own versions (I wish I didn't have to manage my own versions and I've been developing in Python for the last 15 years). The alternative is to ship the runtime, stdlib, and dependencies with each application, but that seems pretty sucky for an interpreted language and especially one that leans so hard on random native extensions (I've seen relatively simple applications pull in 250MB of dependencies in a compressed tarball, never mind runtime and stdlib).

Really I don't think Python (and probably other interpreted languages) is a very well suited for developing end-user applications--although some people have fewer scruples than me about wasting user resources.

Please rest assured that this is not a flamey / troll comment, it's just that your first paragraph to me sounds like "Python is just not worth the trouble" which you kind of allude to in your second paragraph after.

I mean, at what point should we start asking ourselves fundamental questions like "what is so heavenly about Python that we have to carry it everywhere and ensure its installation on every possible platform"?

Example #1: I know a guy who regularly writes scripts in OCaml and laughs at everybody else, saying he gets static types and mega-fast binaries (faster than Golang and slower than Rust) and a very terse syntax (he says it's often as short or shorter than bash even).

Example #2: I recently finally had enough of bash/zsh scripting weirdness for a few tasks I wanted done on my home server so I just learned enough Lua for an hour to make a fairly decent script that did exactly what I expected immediately after I understood a few of Lua's quirks (which are at least consistent and easy to remember, unlike the endless escaping and quoting rules of the scripting shells).

Example #3: I know a few people using a Rust library/tool that makes scripting with it much easier and almost painless. They swear by it.

--

I am not even mad at Python for potentially putting hundreds of megabytes or gigabytes of baggage on people's machines -- if we start having each app carry its own Python runtime. Meh. Many people would never care (me included). I do get ticked off by it being everywhere however; especially having in mind what an awful language for long-term projects it is (said to me by at least 5x senior Python devs at 40+ of age). And how often it messes up with other work you do on your computer, too -- which is the cherry on the cake and has been guilty of tons of broken Debian system-wide upgrades (among many others).

I wish people stopped getting tempted by "easy to start with" languages and just start seeing the forest and not only the trees... :(

I generally agree. I don’t think Python is even “easy to start with” these days. I think it got that reputation in the early 00s, but since then the competition has surpassed it.

Languages like Go are arguably easier to use and getting a basic project up and running and sharing it with others is far easier than with Python. If you really care about a scripting workflow, it’s as easy to do ‘go run main.go’ as it is to do ‘python main.py’ for projects without dependencies and for projects with dependencies it’s a lot easier to just do ‘go run main.go’ than configuring a venv and installing your dependencies and then running ‘python main.py’.

Nit: Ocaml is about on par with Go. Some programs will be faster and others will be slower, but OCaml is very much in the same performance tier as Go, Java, and C# (benchmark game prohibits idiomatic Go code for certain benchmarks so it appears artificially slow).

> … prohibits idiomatic Go…

No it does not.

Per the sibling thread, I think you’re mistaken, but that’s okay. It’s a widely written about topic and anyone can read up on it. We don’t need to hash it out here.

> We don’t need to hash it out here.

We don't: as you have shown absolutely nothing to back up your claim, it can be flatly rejected.

Please show that Go is being made to appear artificially slow, as you claim.

It’s not allowed to preallocate memory in certain benchmarks, so it has to do allocations in a tight loop. Go’s allocations are more expensive by design because idiomatic Go doesn’t create as much garbage as other GC languages.

If you’re here for a language slap fight, you’ll have to find someone else to talk to.

EDIT: good grief, the overwhelming majority of your recent comments are “debates” about the benchmark game. Do you have an alert for references to this phrase or something?

Well tossaway’s account was created 66 days ago so clearly not, and anyway I’m not going to be baited into your holy war.

> They really shouldn’t be bundling any Python version with the OS

I fully agree, Operating Systems bundling Python and Ruby has caused me many headaches in the past and I always perceived it as a downside, never an advantage.

I mean the advantage is that you can run vanilla python on most things by default, which is how Ansible pretty much "just works".

Or rather, people should stop using newest and shiniest version of their interpreted language and stick with what most users have.

Python developers also should stop introducing breaking changes to their language with every release, like it is some toy.

> They really shouldn’t be bundling any Python version with the OS - the user should choose the versions installed.

Why? Having a bundled Python version - any version - means that you can ship Python scripts to others. Not having a Python version bundled means you can't ship Python scripts at all.

Sure, you can still technically ship them, but now you need to tell users "just run this simple scipt!... after you install Python X from place Y, and now you'll have multiple Python versions installed, so here is how to keep them separate, and [...]". Or, you can bundle your script with a whole Python installation!

It’s only really feasible to ship APIs in an end-user operating system that can make forward binary compatibility guarantees.

Python doesn’t make a sufficient guarantee of forward compatibility for either scripts or its C-linkable libraries. This means that, unless an OS vendor does a bunch of extra maintenance work on the Python that they ship, an application built in or making use of the Python shipped in one version of an OS may not run on a subsequent version of that OS.

When I have to install a CLI tool and it's in python I sigh loudly. Happy that many CLIs have moved or are moving to golang

I use pipx for this. It isolates each tool in its own virtualenv. You might still have to install a specific Python version (although the latest 3.X works fine for me), but it takes some of the pain out of the process and makes sure you don't pollute your global/system install.

On a Mac, it could be as simple as (assuming you already use Homebrew):

    brew install python
    brew install pipx
    pipx ensurepath # adds ~/.local/bin to $PATH
    pipx install <tool>
    <tool>

Pipx is really one of the best things that has happened to Python recently. Taking away all the headaches of manually managing venvs. They should integrate it into mainline imo.

That is all nice. And then python updates from 3.9.n to 3.9.n+1 and nothing works anymore.

brew sucks with python.

I've only run into issues like this when upgrading from Python 3.X to 3.Y, which is to be expected, and Homebrew lets you install a specific 3.X (e.g. brew install [email protected]) if you don't need/want be on the latest version.

Thanks, I didn’t know about pipx. If it works as advertised that will be a huge improvement for me!

pipx looks fantastic, thanks for sharing.

As a non python native, I sigh whenever I have to install anything in python. There's always a dependency I can't find and never know why. I never understand why the dependencies aren't just resolved by an application and local to that application. I mostly end up with something working. Sometimes I just create a VM for that single application.

I once really desperately needed a Python tool. Wasted an hour and getting more and more enraged trying to install it, overwriting hell-knows-what and ruining god-knows-how-many other programs in the process. Eventually it did occur to me to just use Docker and that did it. <facepalm>

Every time I am forced to do something with Python I end up hating it. It seems those who write it are just not interested in anyone who doesn't have a complete dev environment setup for it. :(

> It seems those who write it are just not interested in anyone who doesn't have a complete dev env...

And that's probably what makes Python so popular, and so great. I get it. It's attractive to people who aren't developing software for other developers.

But what you described is surely the result of that atmosphere.

> And that's probably what makes Python so popular, and so great. I get it.

I don't follow because that's actually a problem. Imagine having to install Haskell compiler -- exactly one particular version of it with the exact right combination of flags (and there are hundreds of them) -- for each project... while also having the whole thing kind of compile from source.

And that for people who just want to do something like `brew install a_valuable_tool` and have no patience (and certainly not the time) to tinker with Python's dev environment just so they can install a tool they need.

So maybe I am misunderstanding but no, that's not what makes Python popular or great, at all. It makes it a legendary pain for anyone who doesn't code it every day.

If I can't do `brew install your_tool` but have to fiddle with the intricacies of your language's ecosystem then you have failed as a tool author.

Contrast this with Golang tools (disclaimer: I dislike Golang) which in the worst case scenario involve 1-2 commands to download and compile+install the tool but usually don't involve even that because Golang's process is self-contained and that makes it super easy for all package managers to distribute it pre-compiled for their platform.

The distro should be packaging software for end users, relying on the python version they maintain. The problem is that Apple isn’t doing that at all, and brew seems to have issues getting all formulas to converge on known-good dependencies.

Try Anaconda (or miniconda/mamba): like virtualenvs but reproducible, dependency pinning, and a great solver. All in one handy tool

Thank you.

https://xkcd.com/1987/ says it all.

Some of this is brew-specific. The way it handles ownership isn't great.

I'm not sure if it resolves your ownership issues, but I am glad they went with `/opt/homebrew` in ARM Macs to at least keep homebrew in its own directory.

What they should have done from day one. The kludge they used in /usr was broken-by-design.

The way it handles ownership (or at least used to) is completely broken with screwed up stuff happening inside /usr land.

Macports puts everything under /opt/local and you add /opt/local/bin to your PATH.

I've never understood the preference for homebrew, when I moved to MacOS (from Linux) 5 years ago as my working development environment, I read about both and homebrew seemed like a complete kludge, whereas Macports worked the way it should.

SIP just made it even more relevant. Macports deals with specific MacOS related integration (eg Java etc) using the Apple blessed mechanisms while still providing all of the flexibility of the many many packages.

That’s hillarious. It’s why I love Rstudio.. don’t have to deal with any of that as someone who is just trying to prototype and get something to work.

nobody has ever shipped R as part of the base OS

Except Ubuntu DSVM

I feel like in general Python on Mac OS is messy compared to Linux and even Windows.

There is a lot of good things to say about Python which is a fairly nice language but I feel it is messy on every platform. Tooling makes the situation bearable nowadays but that's definitely not one of Python strong point.

But it's so easy to set the path up to python? I hear people complain about this all the time but I never encounter it because I'm careful and I know which interpreter I'm running ALWAYS. It really isn't a hard task to do. Having python in the OS is fine. I wish windows had a default version and I love that it's available on linux in most distros.

Apple ships with a number of utilities installed that require Python, like smtpd.py.

Where does this utility exist? I can't find it on my macOS installation.

A lot (all?) has been cleared out in Monterey. I can’t find smtpd or xattr’s scripts that required the previous version. I think whatever remained before 12.3 were Automator’s py scripts. But I don’t know what scripts those were.

Mine does seem to contain `/usr/bin/smtpd.py` with a modified date of January 22nd. It's an installation that's been updated through a number of versions of macOS, though.

Ah, I see...

It contains the following shebang:

  #!/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python -R

Which continues to work since it's a system framework.

I'm running Monterey on a brand new machine (no updates from older versions of macOS) and mine is in /usr/bin/.

It seems that Spotlight doesn't index that folder, hence why I didn't find it in my quick search.

I see it now, but it uses a shebang that points to the system framework python:

  #!/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python -R

Having some python always installed can be nice as an advanced calculator or short ad-hoc scripts.

My non tech mom needed help to wrestle a large set of semi structured data into a csv to use with Excel. On my dev machine I would have solved the problem in a minute with 5 lines of Python. On her machine, without any coding environment, I was completely stuck. Eventually figured out you can use the developer console in chrome to run some interactive Javascript and problem solved.

Maybe just call it `system-python`.

That’s basically what RHEL does: /usr/libexec/platform-python.

The only reason I'm using python is because it's available on most linuxes and macos by default. If it would be removed from macos, I'd never use it.

MacOS (Certified Unix OS) is bundled with multiple OpenSource projects since it's initial OS X release, just like many other *nix OSes. It's so easy to install your own version and override it that this is not an issue. Simple response... use brew.

PSA: don't use brew to manage your development python, unless you like having the carpet pulled out from under you by brew deciding to change 3.8 to 3.9 on you.

Use brew to install pyenv (or bootstrap it from github). Use pyenv to manage python versions, and then use a virtualenv.

Or conda, if that is your jam.

https://github.com/pyenv/pyenv

Don’t use brew to manage your development anything. Carpet-pulling is pretty much intrinsic to its design.

“hey I upgraded openssl for you and trashed the old one, hope you hadn’t built anything against it!”

Yes, this is what I do now and since moving to pyenv I haven't had any real issues. The only glitch was that I have to upgrade pyenv from 2.2.3 to 2.2.4 and it just tells me 2.2.4 is already installed when I'm running 2.2.3. I'm guessing that's a homebrew thing?

>It's so easy to install your own version and override it that this is not an issue.

It's easy to override it? Really? Don't forget that macOS comes with bash, zsh, ksh, tcsh... You have to override it in each of those!

You can easily set your $PATH so that your preferred tooling is first in the search order.

There's a bunch of places that can still get tripped up:

- scripts with `#!/usr/bin/python` instead of `#!/usr/bin/env python`

- shell scripts which call /usr/bin/python directly instead of the one in your path

- shell scripts which intentionally set a prefix in $PATH to some vendor directory with a custom python, but the vendoring is incomplete and leaky and it implicitly loads packages from your system python anyway

plus a hojillion other things I'm not thinking of...

But scripts calling /usr/bin/python want to use the "system" / "bundled" Python - why would you want those to run with your version of Python???

Because in 120% of situations, that "want" is actually "I had no clue how *nix works and just pasted in the result of `which python` in my system".

Do you think the same is true of #!/bin/bash? Why not?

GUI apps also have their own special $PATH set via launchctl.

macOS*

I don’t get your point. Are you agreeing?

you should absolutely avoid overriding the default python installation if you're on a distribution that ships python2 by default in particular, because that is a good way to kill a good chunk of your system.

Everyone should depend on “python3” or “python2”, never just “python”. They aren’t really the same language. python3 can’t run python2 scripts, nor vice versa, so no matter what you expected there will be systems on which you won’t get it.

100% this.

Thank you, I struggled with that problem for years. Happy that I am not the only one.