Perhaps in situations like this Homebrew could alert the user "Hey, this package hasn't been updated in a while and 40 other packages depend on it so this is going to take a while. Press enter to confirm or cancel and use an alternative method (HOMEBREW_NO_INSTALL_UPGRADE) to just upgrade the one package. Caveat - this may break things."
My only point (and a source of confusion) here is that I never anticipated the cascading effect that installing a single package can have on seemingly unrelated parts of the system. I've been using homebrew for many years and as far as I remember, the only time anything broke like this was when I explicitly ran update/upgrade.
I'm going to try HOMEBREW_NO_INSTALL_UPGRADE/HOMEBREW_NO_AUTO_UPDATE — it does seem like a more intuitive default, but again, I don't know anything about how homebrew works under the hood.
Thanks again!
Personally, I have moved to downloading large things (Python, Postgres by your example) as standalone apps, and only use Homebrew for small libraries and CLI tools. I think that makes more sense.
You can download all of the Mac Python versions you want from python.org and you can get Postgres from postgresapp.com which provides a nice toolbar interface to start and shutdown servers of various versions.
With homebrew, I never know how many packages will be updated and how long it will take, so sometimes when I just need a small utility, suddenly it'll update a bunch of packages and it takes 45 minutes.
It's a UI problem.
I'd add that script-y usage of brew already can block on user input (e.g. if a cask asks for sudo), so adding a go/no-go check probably only breaks the most determined automators, while the benefits accrue to a wide swath of users (both new and fat-fingered).
Regardless, thanks for all the work you (and the community!) do on Homebrew! I wouldn't like macOS half as much (or have bought half as many) without it.
brew outdated will list outdated app/dependencies that need to be updated. If you want to know the app's dependencies, use the info argument. For example brew info ffmpeg will list the information about FFmpeg and list the dependencies that it uses along with status indication if the dependencies is installed or not.
Am I missing something here? I am not sure if it is a UI problem since Homebrew have those functions with a few arguments here and there. Look like you are using a utility that have so much dependencies that it needs or you have habit of using brew upgrade which will prompt to upgrade everything expect pinned apps/dependencies.
Yes. The principle of least astonishment.
If Postgres has an unversioned dependency to PackageX, the only time I might expect Postgres to be updated automatically is if I update PackageX across a major version boundary, risking breaking changes. Even then, I'm not sure I'd want that behavior by default - I'd at least like to be prompted - since often this still won't break anything, or it will likely cause less damage than updating every package with a dependency to PackageX. And if Postgres breaks after I update PackageX I can easily reason about what just happened. The alternative can be incredibly hard to troubleshoot because so many changes have occurred.
Overall, Homebrew feels far too aggressive here, and I think the result, for people who have some understanding of Homebrew behavior, is that they're afraid to touch anything.
Homebrew doesn't meaningfully have versioned or unversioned dependencies, because it doesn't incorporate any notion of version into its dependency solver. When you declare a dependency on `[email protected]`, you're not telling `brew` that ‘I need a copy of the package `python` which satisfies the constraint “version is 3.9.x”.’ Homebrew just treats `[email protected]` as the name of a package which is a whole copy of python, and it can exist alongside other, similar packages like `[email protected]` and `[email protected]`.
Homebrew only incorporates this kind of `@` syntax for select packages of which maintainers have elected to maintain multiple versions. By my count, there are 81 such packages in homebrew-core, and the following is a complete list of all of those which appear in a `depends_on` declaration:
antlr
atkmm
autoconf
cairomm
db
elasticsearch
erlang
gcc
ghc
glibmm
go
gradle
guile
hdf5
headers
helm
isl
libfuse
llvm
lua
mbedtls
mysql
node
numpy
openexr
openjdk
openssl
pangomm
php
proj
pyqt
pyside
python
qt
ruby
swig
synth
tbb
vtk
wxwidgets
So when you look at a Postgres formula in the Homebrew repos and see that it declares an ‘unversioned’ dependency on openldap, for example[1], that does not mean that the formula author has declared ‘any version is fine’, i.e., it does not mean that the dependency is unversioned. It just means that the dependency's version constraints are totally unspecified except implicitly as ‘the only version of that package which is simultaneously present in the glorious monorepo of build recipes’.This is a feature it shares with some source-based package management systems, like Nix and Guix. Those package managers solve this problem by basically vendorizing everything, so installing a new package never upgrades anything else you have installed.
Other source-based package management systems, like Portage and Pkgsrc, have extended their model to include some handling of version constraints, but they also sometimes declare dependencies without specifying a version. That seems dubious to me, since in reality there pretty much always are version constraints that ought to be specified there, but maybe those two projects are mature enough that they've hammered out the problematic cases by now.
—
https://github.com/Homebrew/homebrew-core/blob/master/Formul...
But I think it's unfair and lazy to say that Homebrew's design defects are just there because its author is stupid or something like that. That Homebrew's (massive) success is grounded in virtues other than fundamentals like dependency management doesn't mean that Max Howell has nothing to offer or couldn't be an effective part of a team at Google or anywhere else.
I imagine his willingness to just forge ahead with a simple solution when he notices that existing tools aren't serving him well means that he doesn't often get totally stuck, or succumb to paralysis by analysis, for example. He might be great to pair with if you tend to overthink things or get stuck, and I bet he'd be a very helpful critic and contributor when it comes to the user experience of internal CLI tools.
There's no need to shit on the guy. He's been very honest and clear about Homebrew's strengths and weaknesses: https://www.quora.com/Whats-the-logic-behind-Google-rejectin...
That said, I did notice recently the same effects as the OP. I.e. I try to install something small and it depends on something like python, which triggers "upgrade the whole universe" loop, which breaks a ton of things (e.g. python upgrade doesn't carry over installed modules, but does change current version - which breaks many local python scripts since their dependency modules are now in the old version). Auto-cleanup tends to break a lot of things too - especially bad for commonly-used things like openssl or ICU which commonly are linked as shared libs.
I think there are settings for changing all of this, but you only find out once you've already been bit. Maybe it's worth considering making it somehow explicit - e.g. ask the user "do you really want to upgrade the whole universe" before going in? In many cases, I'm completely fine with using python 3.7 even if python 3.10 is already out. And most of the tools that depend on python would be fine with it too. It would be nicer if low-friction mode would be the default.
> Now, Homebrew has a choice. Either we upgrade _everything_ that depends on `[email protected]` that you have installed or we knowingly break some of the things you have installed that depend on `[email protected]`. We choose the safer option by default.
A look at the formulae in homebrew-core shows that Homebrew basically doesn't consider versions in its dependency management at all, so formula are just using the ‘@x.y.z’ in place of all manner of version constraints. So you get wild stuff like this line here[1]:
depends_on "curl" # curl >= 7.55 is required
wat.Of course the recursive update policy is going to be weirdly painful for some users! Homebrew doesn't even attempt to encode some very basic aspects of dependency, so now (a) you can't know what you're going to break if you do upgrade a common dependency and (b) you can't know when you can afford not to upgrade a dependency shared by a package you're about to install and the already-installed system. The choice you outline above is one that is not faced by most package managers, because they don't make this mistake. The naive wheel reinvention with Homebrew is so eternally disappointing, and it explains a lot of the pain users experience with it.
> Homebrew upgrades dependencies and dependents of those dependencies (which, admittedly, can feel like unrelated)
One relatively non-disruptive thing you might be able to do to make this behavior less surprising to users is (offer a way to?) print the dependency tree for package installations/upgrades that pull in upgrades of their ‘siblings’. You'd probably want to just do it in a textual way, but this project seems to model the kind of logic you'd want for printing dependency trees with Homebrew as it exists.[2]
—
1: https://github.com/Homebrew/homebrew-core/blob/master/Formul...
2: https://github.com/martido/homebrew-graph/blob/master/cmd/br...
What’s the difference in what Homebrew is doing from what normally happens with something like Pacman or Apt? My impression is that “core” binaries, like Python might be, don’t get updated as often as they would on Homebrew, and perhaps those dependency chains are less frequent. Is it because (e.g.) python versions change rarely on linux package managers? Or something about how the binaries are compiled?
Homebrew is a source-based package manager, where packages are built from a single shared source of build recipes, and distribution of software happens by distributing the whole collection of recipes. Support for a binary cache is added through the identification of build recipes with the hashes of archives of binaries.
APT is what's called a binary package management system. While build recipes can be stored in a central repository, they don't have to be, as binary package management systems typically identify two kinds of packages: source packages and binary packages, which can be distributed separately. (Some repositories may not even include source packages.) Source packages contain metadata like build recipes, but things like archives of the upstream source code, patches, and sometimes helper scripts that might be usd in building. Binary packages include the actual built artifacts; they're installable packages.
In the source-based package management world, if packages can be added to the main collection through external sources, this is typically done by pretending that those packages were embedded in (some particular version of) the main repository of build recipes. Collections of packages that are ‘injected’ into the main package collection like this are typically called ‘overlays’.
In binary package management systems, an external package is not expected to have access to some copy of a complete collection of build recipes from some ‘master’ repository. Instead, source packages are distributed individually using the same mechanisms as binary packages. This means that the global namespace of terms which refer to packages can't be navigated implicitly at build time like on source-based systems, the identification of actual source code and build instructions with some package name happens both at build time and at install time. And in both cases, explicit metadata (package names, package versions, equivalency classes, lists of conflicting packages, etc.) is used to determine what versions of dependencies get pulled in.
As a result, binary package management systems always include dependency solving at install time, and usually include some explicit version constraint for every single dependency declared in a package's build recipe. This means they have the resources to look at what's going to be installed, what's already installed, and say ‘oh, the new installee requires Python <= 3.10.4, and since your installed packages together only need Python >= 3.10.2 and you already have python-3.10.3 installed, we don't have to touch Python to install this new package unless you ask us to’.
Pacman is somewhat atypical. The build system is essentially source-based and monolithic (there's no source package format) and the AUR is essentially an overlay, but the high-level CLI tool acts like a binary packager and uses the language of repositories to describe dealing with multiple package sources. Pacman's dependency resolution behavior is also unusually unreliable, and probably closer to what `brew` does than what `apt` does. Pacman has the resources to describe and test version constraints, both in its build system and in its CLI tools, but they're underutilized in a way that's more typical of source-based package management systems than binary ones. Homebrew is similar in that such constraints are radically underutilized (they're totally absent).
The main difference between Homebrew's behavior and Pacman's in this instance is that by default, just `pacman -S` doesn't update pacman's collection of package definitions (pacman -Sy does) and Homebrew always tries to install from the latest build recipes.
I imagine that Pacman's capacity for handling version constraints means that the story with `pacman` will usually be a bit better than with `brew`, but that otherwise `brew install` will be similar to `pacman -Sy` and `HOMEBREW_NO_INSTALL_UPGRADE=1 brew install` will be similar to `pacman -S`. (Pacman also sometimes makes the opposite tradeoff that Howell describes for Homebrew sometimes; it's not unheard of for installing new software or updating software using Pacman to break installed software.)
The irony is that poster was trying to install a solution to this very problem (virtualenv).
The broader problem is that, at this point, python is effectively its own OS. Coupled with macOS being schizophrenic between "We want to provide a posix environment for users" and "No user should ever look under the hood, so no need to tell them about all the duct taped bits."
Which brew attempts to bridge. Admirably, but never perfectly.
Obligatory: https://xkcd.com/1987/
As a result, Python packaging has an utterly insane design that it can't evolve its way out of.
Problems that have proven more or less intractable for resolution by evolutionary change:
- total lack of static metadata— you must download a source package and attempt to build it in order to determine what its actual dependencies are
- relatedly, package setup scripts are way too powerful and can do literally anything, which makes automating Python packaging a nightmare
- any possible vendorization is deeply limited by the fact that Python processes can't include multiple versions of a single library
- the behavior of all Python packaging tools relies on the implicit state of PyPI, forcing a high degree of non-determinism unless you go to great lengths to take snapshots of PyPI
- native dependencies (i.e., dependencies on C libraries) are either just totally undeclared, or packaged abuse setup.py in order to manually compile them in an ad-hoc, unmanaged fashion
There's no way out of this that doesn't involve breaking things for Python developers who are, whether out of shortsightedness or resigned pragmatism, relying on or working around the many, various bad practices that the Python packaging 'system' currently allows.This is why programmatically generating Python packages for general-purpose package managers like Homebrew ranges (depending on the resources made available by the package manager's design) ranges from impossible to stupidly complex and computationally expensive to kinda works but has no hope of reliability/correctness/completeness.
What's a good example of prior art that Python might have considered in its design? This whole discussion is super interesting to me and I'd like to read more on better examples of dependency management.
I'll answer this before we do some Python history. Since there are a range of paradigms, if you really want a good sense of the main ways things are done and the tradeoffs involved in mature systems, take a look at: dnf/RPM as it's done on current Fedora, Gentoo Portage, Guix, Cargo from Rust, Flatpak, macOS' .apps, and Distri. Each of those represents a different approach.
(You could get away with just looking at dnf and Portage, which also represent modern iterations of the paradigms Python could have looked to in the early days of its own software distribution tooling, if not the earliest days of the language itself.)
SOME PYTHON HISTORY
That's a great question, in part because Python has been around so long! It's easy to collapse the past in a way that's uncharitable to projects like Python, so let me expand on this a little and cross-reference some dates to get it into perspective for both of us.
The best examples to consider today for dependency management are probably Linux distro package managers, *BSD ports systems, and Rust's `cargo`. But Python is older than Linux or any of the major surviving free Unices! So one legitimate difficulty Python has faced here is its age— what Python folks could have looked to for inspiration in 1991 was, compared to today, extremely limited and much less publicly available. They were not positioned, as the designers of Rust were, to really think through how code in the language would be distributed on day zero, before the language would see any real adoption.
There have, however, been opportunities for the Python community to rethink things in light of the work and experience of projects (like Linux distros or ports systems) where dependency management is a very central problem. There have also been occasions on which some breakage was already expected or permitted, and such a change might have been feasible. But I think the focus has been elsewhere, and the Python community and its leadership have had to face the very real risk that too many breaking changes in their language's tooling could result in shrinking or dividing the community in a way that Rust hasn't yet. So the issue isn't some unkind nonsense like ‘Pythonistas are idiots’ or whatever.
That said, by the time distutils rolled out as part of Python's standard library in 2000, apt, Debian's second-gen high-level package management tool was out and had undergone a couple years of battle testing. The *BSD ports systems had also been around for years by then, and Gentoo would hit the scene leveraging a similar approach to build a whole Linux distro that same year. I'm not sure what Linux package management tools were like, but that would certainly have been enough to tell already that static metadata for source packages was a must (and so something like setup.py is not appropriate).
Similarly with the creation of setuptools in 2004— by then there were lots of examples of mature package managers in two major paradigms: source-based (like Homebrew), and binary (like most Linux distro package managers). Similarly again with the rollout of Python 3.0 in 2008. When pipenv came out in 2018, the answer to questions like ‘why does pipenv take so long to freeze dependencies into a lock file’ (‘it has to download every package and pretend to try to build them’) revealed that the Python packaging world had failed to meaningfully incorporate lessons on package management from the outside world in the intervening 10 years.
You can watch/listen to a fairly non-technical overview of the history of Python packaging here: https://www.youtube.com/watch?v=AQsZsgJ30AE
Homebrew was a much later arrival, so its case is a little more baffling. But some of its choices that now seem naive or questionable should also be credited with its widespread adoption. For example, the choice to depend thoroughly on the underlying macOS system instead of isolating dependencies makes Homebrew pretty brittle in the face of upgrades between macOS releases, but it also made it much more convenient for getting things installed quickly from a blank slate state than other source-based package managers (i.e., MacPorts) for macOS at the time, since it meant less compiling from source. Similarly, while the simplicity of Homebrew's build recipe DSL is a current source of problems, that same simplicity (paired with embedding it in a popular, interpreted language) has made Homebrew feel approachable and helped to draw in the huge number of contributors that have worked on Homebrew's formulae.
To wit, only people interested in new languages spend lots of effort building new languages (and their associated ecosystems).
People who have been using an existing language for years, long enough to internalize the design decisions and flaws, have little incentive to stop using something they're an expert at, to build something new.
Ergo, the person you really want designing a language and ecosystem is someone who's used existing art enough to be an expert, then decides to give it all up, start from scratch, and do something harder.
There aren't many of those people.
The insight that the technical is not entirely separable from the social is an important one, imo.
AFAIK homebrew cannot automatically modify $PATH before starting a program such that $PATH points to the correct dependencies for that program. If you have multiple versions of some dependency installed, it is up to you to run `brew switch` before running your program to manually set the correct versions for each dependency.
Thanks for all the work you put in homebrew, I really appreciate it. Just upgraded to Monte Rey a few days ago and I was kind of fearful that some of the things that I needed where not going to work... nop, every singe package that I've needed so far was installed successfully.
Thanks!
My life became much better after setting this.
I wouldn't be averse to a HOMEBREW_AUTO_UPDATE_INTERVAL=180d either, or HOMEBREW_AUTO_UPDATE_ASK=y .
I think we need a new package manager since its not the fault of homebrew that many packages have hardcoded stuff in there.
Anyway, I just wanted to say you guys are doing a great job!
---
Homebrew upgrades dependencies and dependents of those dependencies (which, admittedly, can feel like unrelated) on installation and upgrade. As mentioned in other comments, you can customise this behaviour with `HOMEBREW_NO_INSTALL_UPGRADE` or `HOMEBREW_NO_AUTO_UPDATE`.
Homebrew does this because the alternative is sometimes breaking things. An example:
- you want to install `virtualenv` that depends on `[email protected]`
- the binary package for `virtualenv` you want requires the newest `[email protected]`
- this upgrades `[email protected]` on installation
Now, Homebrew has a choice. Either we upgrade _everything_ that depends on `[email protected]` that you have installed or we knowingly break some of the things you have installed that depend on `[email protected]`. We choose the safer option by default.
The more time left between updating/upgrading, the more likely to have more dependencies updated which requires more dependents to be updated.
---
Regardless, I appreciate this is a problem and we're still figuring out potential solutions for this problem. A reminder that we're a volunteer run project so it's not always as easy as we'd like it to be to get these changes out quickly.