All the projects you listed (Pandas, PyTorch, SKLearn) have a huge list of compatibility issues, on top of compatibility issues of Python itself. And I know this because I have to support various research projects, which typically use this stuff. My estimate is that the shelf-life of a project using just these three will not exceed 3 years in >90% of cases. My typical situation with these projects is that I'm called in because some former PhD student, who graduated a year or two ago left a project that now another student is picking up. And nothing works. Results cannot be reproduced. The dependencies cannot be installed. And so on.
And then, depending of the amount of effort I'm allowed to put into fixing the project, I end up either crafting some sort of a spec that tries to find a set of versions of used libraries that seem to make the project work again. Or, more commonly, have to vendor some code and add fixes on top to make the packages fit together at least somehow. Sometimes I even end up repackaging someone else's code (a dependency), where the version used before cannot be found anymore, or I cannot figure out the combination of version that could possibly make the program work.
I hear that it's worse in JavaScript world. And, my experience with LaTeX packages has also been marred by similar problems (but not anywhere near the extent of how bad it is in Python).
On the other hand, I had to build absolutely atrocious (math operations on a mix of signed and unsigned ints, modifying const char etc.) C code (pre-K&R, the kind where they write function argument types after parenthesis), and it was much easier to deal with than any "modern" Python (i.e. 3-4 years old).
If anything, Python is the poster child of the problem of unreliability of third-party dependencies.
As someone who works in both academia (currently a PhD student!) and industry (at the aforementioned startup), it just seems obvious to me that the situation you are describing is less of an innate issue with incorporating dependencies and more an issue with the lack of training on environment management / containerization and lack of experience with software development in general that you see in academic projects. In a relatively competent professional shop, these sorts of issues in my experience crop up occasionally but there are so many mechanisms available to (a) mitigate them from appearing in the first place and (b) have rough plans or strategies in place ahead of time if they do crop up and (c) continuously address them when they do appear, which prevents it from ever blowing up into a true dependency hell. It seems like being judicious about when you decide to use dependencies and how you manage them is sufficient, as opposed to actively avoiding them altogether...
Anyways to me it definitely is the casethat Python has loads of footguns/risks in re: dependencies when someone is developing without thinking about dependency management/long term planning, which is usually the case with academic projects, but I don't think these dangers are so bad when the user is actively aware of them and plans for them - or not bad enough to make the risks outweigh the rewards of adopting the dependencies... it definitely means that the ecosystem is elevating risk levels, but I would also think that there are organizational/management issues which contribute significantly to (and compound with) the issues you've described in an academic context.
> My estimate is that the shelf-life of a project using just these three will not exceed 3 years in >90% of cases.
That seems reasonable; it's problematic from the perspective of the replication crisis in the sciences, but from a different perspective - if the code/project has no users or active developers for 3 years, at what point does it just become cruft? Is it okay to let code die? I don't love this view, but my PI strongly feels that no matter how excellently or poorly engineered a GitHub repository is, it doesn't matter from the perspective of sciences - the only thing that matters is if (a) the methodology is explained thoroughly enough in the literature you produce (in which case the repo can be trash and unrunnable in a few years - if the method is useful enough it will be copied and expanded on etc by others naturally or just completely independently, and 3-5 year old methods are out of date anyways) or (b) it goes on to be actively used by you, your peers, or people in industry, in which case the literature does not matter (but by definition the repo is not trash as it is in continuous use/active development/evolution). I think their are some flaws in this view but I also see merit in it.
Anyways, having said all of this, I hate it when one of my peers at the startup I work with wants to pull in some dependency for a relatively straightforward search algorithm they don't understand when I think it would be easier to just implement it ourselves, so I'm contradicting myself :)
At the time the hardware cannot possibly support it. So, we are talking about something like 50 years. For more high-level languages the deadline should extend further (i.e. languages like C should, in principle, have shorter shelf life because they have stronger dependency on the hardware they run on). This is how, ideally it should work. I.e. there's no reason any Python code should be dead by now: Python simply hadn't been around long enough for that to become a real, impossible to solve issue. Ironically, C programs have far better shelf life than Python. About an order of magnitude better.
Second deadline is when businesses will stop using project's code if it doesn't have enough shelf life. This is somewhere between 10 and 5 years. Government / military will want the longer shelf life (and usually have enough bargaining power to compel the developers), while the rest of the industry will suck it up and accept the inevitable death of their software. But, at some point, it no longer makes sense to sacrifice the profits just to keep up with the trends. Eg. you can still find businesses that haven't upgraded from CentOS 7 (released in 2014), and are doing OK. A lot of military software uses 90s or even 80s tech. Banking is also a special case here, as well as some embedded systems that can be many decades old eg. railway networks.
I wish that anything below that threshold wasn't commercially viable... but, obviously, a lot of businesses find enough resources to upgrade more frequently. I feel like these businesses make a disservice to the broader community of users depending on the same technology. In some ways, their ability to upgrade more often is also a way to beat the competition. Unfortunately, more often, it's not just the competition that suffers from this, but also businesses in completely unrelated fields.
----
Now, if talking about Python specifically, there's quite a bit of difference based on particular area the code is written for. Web and data science are the areas with the worst shelf life, and require the most effort to simply stand still. These are the areas where projects typically have more dependencies, attract a lot of attention (and so there's more drive to iterate faster on those dependencies).
But, on Python's side, there are plenty of bad things happening regardless of what individual teams can do. For example, when talking about support of software like operating systems, this includes support of a large number of individual (optional) components that need to work together. I.e. the LTS Linux distribution will have a set of packages that are (allegedly) tested to work together, and if you want to have a reliable baseline for your own software, you just have to draw from a particular distribution and you should be fine. PyPI, or more broadly, Python community doesn't have that kind of concept. There's no set of dependencies that can be guaranteed for any length of time at any time. Anaconda tried to be more like that, but they were overpowered by the users' desire to get new things out of bound.
Another problem with how Python versions packages is that it imposes virtually no restrictions on publishing. If a package author wants to upgrade ten major package versions in a day, there's nobody to tell it to stop. Want to publish an older version than the last one published? -- sure, go ahead. And, of course, on top of it, interface breakage in versions where that's not supposed to happen etc. The poor stewardship of PyPI is a self-fulfilling prophecy: package maintainers realize that the versioning is, essentially, broken in Python, and in order to prevent their packages from breaking, they narrow down the range of acceptable dependency versions as much as possible (and often only accept the single version of the dependency they tested with). This, in turn, results in a situation where upgrades are impossible without massive breakage: incremental upgrades become impossible, because no part of the dependency tree can be upgraded independently. On the other hand, if a single dependency in this tree goes stale, the whole tree is stale.
Another factor to this is that dependencies which accept a wider range of transitive dependencies are harder to install using currently available package managers / installers. I.e. because pip / conda will spend a lot more time trying to find a solution for dependency graph with more potential candidates (due to more loose version requirements), developers are prone to try and expedite the process by narrowing down version constrains. This is especially painful in conda world, where the solver can take many hours (or days in pathological cases) to solve such installations. And so developers have to choose between being more friendly to the user (and supporting multiple transitive dependency versions) or being more hostile to the user (by narrowing down supported versions) while speeding up their deployment.
Yet another factor is the design of Python module system that is unaware of module versions. I.e. if only it was possible to specify the version of the imported module at import time, a lot of problems would've gone away. There would've been no need for virtual environments. Subsequently, there'd be more incentive to ensure compatibility of various independent libraries and programs, which would put pressure on package maintainers to dedicate more time to integration testing and slow down the release cycle.
I've lost count of the number of times runtime behavior and performance issues have been traced back to (for us) far-from-optimal design choices in third-party libraries that probably seemed like reasonable choices to the library authors. Particularly in open source, you see remarkably little investment in performance engineering or scalability, so it is always straightforward to write your own implementation that is many times faster on the same hardware. Even basic things like popular data format conversion libraries are surprisingly consistently suboptimal. In most cases, the kinds of changes we'd want to make cannot be upstreamed to that library; most open source libraries heavily optimize for simplicity and the widest set of users to the exclusion of maximum performance. In an age when hardware can deliver >100 GB/s of throughput, many popular software libraries deliver <1 GB/s per core even though far better can be achieved with a modicum of diligence.
Another issue is the impracticality of threading fine-grained metrics and observability through a hodgepodge of libraries that were never designed to be used that way or to work together. This is a big deal if you want to heavily automate operations.
There is also a real issue with supply chain risk in terms of both quality and security. The vast majority of open source libraries are implemented to a lower standard of quality and security than the rest of the code base. And those libraries are often not really designed to be testable to the extent that would give similar confidence in the implementation quality. Combine this with the reality that the software needs to be deployable in environments that require higher than average assurance makes this a business risk.
As for the manpower cost, it has been low in practice. An engineer-month here, a few engineer-months there, with the benefit of better performance, more optionality, and much easier maintainability. In most cases, we would only use a tiny subset of library functionality regardless, so it adds a large surface area for a narrow benefit. Much of this code is also reusable. The existing open source libraries are very useful here because they provide something to measure against.
I also like that this forces a discussion of actual requirements for that functionality. Far too much software makes their requirements whatever a library offers.
I arrived at this point gradually over many, many years. The practical maintainability and opportunity for optimization have been key. In far too many cases, it turns out that a library dependency could be replaced with a couple hundred lines of code that actually worked better and which matched the design and style of the rest of the code base. There is a learning curve but once you've written a common class of library once, it is straightforward to do it again.
> Particularly in open source, you see remarkably little investment in performance engineering or scalability, so it is always straightforward to write your own implementation that is many times faster on the same hardware. Even basic things like popular data format conversion libraries are surprisingly consistently suboptimal. In most cases, the kinds of changes we'd want to make cannot be upstreamed to that library; most open source libraries heavily optimize for simplicity and the widest set of users to the exclusion of maximum performance.
Yeah, I have even encountered this myself once or twice - i.e. re-implementing some optimization/search algos, sampling processes, or numerical simulation methods to be more tightly coupled to our requirements/context for great performance gain - despite not being in a context where heavy performance engineering has become necessary yet. This obviously overlaps with your point about using only a tiny subset of library functionality.
> Another issue is the impracticality of threading fine-grained metrics and observability through a hodgepodge of libraries that were never designed to be used that way or to work together. This is a big deal if you want to heavily automate operations.
Yes, I personally have not ever run in to this because of the relatively small scales I am operating at but makes perfect sense. Same goes for the security concerns. Obviously these are more serious concerns for larger and longer timescale projects (as discussed in the original blog post).
> engineer-month here, a few engineer-months there
I've only worked in contexts where there are no more than 5 simultaneous active devs on a project, so the long-term investment for longer term gains is probably in higher tension with the short term needs. obvious red flag for increasing technical debt/deferred maintenance/downstream complexity, but still can be difficult to justify diverting 20% of team's total productivity for a long term investment when there are short term demands. But always good to be reminded that those long term investments shouldn't be rejected outright and should be more seriously considered.
> I also like that this forces a discussion of actual requirements for that functionality. Far too much software makes their requirements whatever a library offers. I arrived at this point gradually over many, many years. The practical maintainability and opportunity for optimization have been key. In far too many cases, it turns out that a library dependency could be replaced with a couple hundred lines of code that actually worked better and which matched the design and style of the rest of the code base. There is a learning curve but once you've written a common class of library once, it is straightforward to do it again.
This is a really excellent and concise way of summarizing it. I'm definitely going to try to push this with my team more often and now have a great resource for how to articulate it. Thanks again for taking the time to explain your perspective.
What kind of software development do you do? As someone who is in ML/data science and works primarily in python, occasionally TS/React, at a small company (about 5 FTE, one round of funding), that sentence is so completely foreign to me. Again, this might just be because I am working primarily on ML systems in Python on relatively small teams (also in academia at the same time), where the most basic building blocks like Pandas (or Polars), PyTorch, SKLearn, etc are so essential and so it’s pretty hard to imagine ever being at a scale where they would be feasible or worth it to replace with our own code… and anyways if support for libraries like those collapses, the only explanation is that the entire landscape has shifted so much that it’s hard to imagine the underlying tech of the company not also having naturally shifted with it…
Maybe I am misunderstanding how you are using “3rd party,” or maybe you are just operating in a completely different staffing and financial context which I don’t have experience with.
Just curious for more details about what kind of world you operate in where that philosophy is viable!