What kind of software development do you do? As someone who is in ML/data science and works primarily in python, occasionally TS/React, at a small company (about 5 FTE, one round of funding), that sentence is so completely foreign to me. Again, this might just be because I am working primarily on ML systems in Python on relatively small teams (also in academia at the same time), where the most basic building blocks like Pandas (or Polars), PyTorch, SKLearn, etc are so essential and so it’s pretty hard to imagine ever being at a scale where they would be feasible or worth it to replace with our own code… and anyways if support for libraries like those collapses, the only explanation is that the entire landscape has shifted so much that it’s hard to imagine the underlying tech of the company not also having naturally shifted with it…
Maybe I am misunderstanding how you are using “3rd party,” or maybe you are just operating in a completely different staffing and financial context which I don’t have experience with.
Just curious for more details about what kind of world you operate in where that philosophy is viable!
All the projects you listed (Pandas, PyTorch, SKLearn) have a huge list of compatibility issues, on top of compatibility issues of Python itself. And I know this because I have to support various research projects, which typically use this stuff. My estimate is that the shelf-life of a project using just these three will not exceed 3 years in >90% of cases. My typical situation with these projects is that I'm called in because some former PhD student, who graduated a year or two ago left a project that now another student is picking up. And nothing works. Results cannot be reproduced. The dependencies cannot be installed. And so on.
And then, depending of the amount of effort I'm allowed to put into fixing the project, I end up either crafting some sort of a spec that tries to find a set of versions of used libraries that seem to make the project work again. Or, more commonly, have to vendor some code and add fixes on top to make the packages fit together at least somehow. Sometimes I even end up repackaging someone else's code (a dependency), where the version used before cannot be found anymore, or I cannot figure out the combination of version that could possibly make the program work.
I hear that it's worse in JavaScript world. And, my experience with LaTeX packages has also been marred by similar problems (but not anywhere near the extent of how bad it is in Python).
On the other hand, I had to build absolutely atrocious (math operations on a mix of signed and unsigned ints, modifying const char etc.) C code (pre-K&R, the kind where they write function argument types after parenthesis), and it was much easier to deal with than any "modern" Python (i.e. 3-4 years old).
If anything, Python is the poster child of the problem of unreliability of third-party dependencies.
As someone who works in both academia (currently a PhD student!) and industry (at the aforementioned startup), it just seems obvious to me that the situation you are describing is less of an innate issue with incorporating dependencies and more an issue with the lack of training on environment management / containerization and lack of experience with software development in general that you see in academic projects. In a relatively competent professional shop, these sorts of issues in my experience crop up occasionally but there are so many mechanisms available to (a) mitigate them from appearing in the first place and (b) have rough plans or strategies in place ahead of time if they do crop up and (c) continuously address them when they do appear, which prevents it from ever blowing up into a true dependency hell. It seems like being judicious about when you decide to use dependencies and how you manage them is sufficient, as opposed to actively avoiding them altogether...
Anyways to me it definitely is the casethat Python has loads of footguns/risks in re: dependencies when someone is developing without thinking about dependency management/long term planning, which is usually the case with academic projects, but I don't think these dangers are so bad when the user is actively aware of them and plans for them - or not bad enough to make the risks outweigh the rewards of adopting the dependencies... it definitely means that the ecosystem is elevating risk levels, but I would also think that there are organizational/management issues which contribute significantly to (and compound with) the issues you've described in an academic context.
> My estimate is that the shelf-life of a project using just these three will not exceed 3 years in >90% of cases.
That seems reasonable; it's problematic from the perspective of the replication crisis in the sciences, but from a different perspective - if the code/project has no users or active developers for 3 years, at what point does it just become cruft? Is it okay to let code die? I don't love this view, but my PI strongly feels that no matter how excellently or poorly engineered a GitHub repository is, it doesn't matter from the perspective of sciences - the only thing that matters is if (a) the methodology is explained thoroughly enough in the literature you produce (in which case the repo can be trash and unrunnable in a few years - if the method is useful enough it will be copied and expanded on etc by others naturally or just completely independently, and 3-5 year old methods are out of date anyways) or (b) it goes on to be actively used by you, your peers, or people in industry, in which case the literature does not matter (but by definition the repo is not trash as it is in continuous use/active development/evolution). I think their are some flaws in this view but I also see merit in it.
Anyways, having said all of this, I hate it when one of my peers at the startup I work with wants to pull in some dependency for a relatively straightforward search algorithm they don't understand when I think it would be easier to just implement it ourselves, so I'm contradicting myself :)
At the time the hardware cannot possibly support it. So, we are talking about something like 50 years. For more high-level languages the deadline should extend further (i.e. languages like C should, in principle, have shorter shelf life because they have stronger dependency on the hardware they run on). This is how, ideally it should work. I.e. there's no reason any Python code should be dead by now: Python simply hadn't been around long enough for that to become a real, impossible to solve issue. Ironically, C programs have far better shelf life than Python. About an order of magnitude better.
Second deadline is when businesses will stop using project's code if it doesn't have enough shelf life. This is somewhere between 10 and 5 years. Government / military will want the longer shelf life (and usually have enough bargaining power to compel the developers), while the rest of the industry will suck it up and accept the inevitable death of their software. But, at some point, it no longer makes sense to sacrifice the profits just to keep up with the trends. Eg. you can still find businesses that haven't upgraded from CentOS 7 (released in 2014), and are doing OK. A lot of military software uses 90s or even 80s tech. Banking is also a special case here, as well as some embedded systems that can be many decades old eg. railway networks.
I wish that anything below that threshold wasn't commercially viable... but, obviously, a lot of businesses find enough resources to upgrade more frequently. I feel like these businesses make a disservice to the broader community of users depending on the same technology. In some ways, their ability to upgrade more often is also a way to beat the competition. Unfortunately, more often, it's not just the competition that suffers from this, but also businesses in completely unrelated fields.
----
Now, if talking about Python specifically, there's quite a bit of difference based on particular area the code is written for. Web and data science are the areas with the worst shelf life, and require the most effort to simply stand still. These are the areas where projects typically have more dependencies, attract a lot of attention (and so there's more drive to iterate faster on those dependencies).
But, on Python's side, there are plenty of bad things happening regardless of what individual teams can do. For example, when talking about support of software like operating systems, this includes support of a large number of individual (optional) components that need to work together. I.e. the LTS Linux distribution will have a set of packages that are (allegedly) tested to work together, and if you want to have a reliable baseline for your own software, you just have to draw from a particular distribution and you should be fine. PyPI, or more broadly, Python community doesn't have that kind of concept. There's no set of dependencies that can be guaranteed for any length of time at any time. Anaconda tried to be more like that, but they were overpowered by the users' desire to get new things out of bound.
Another problem with how Python versions packages is that it imposes virtually no restrictions on publishing. If a package author wants to upgrade ten major package versions in a day, there's nobody to tell it to stop. Want to publish an older version than the last one published? -- sure, go ahead. And, of course, on top of it, interface breakage in versions where that's not supposed to happen etc. The poor stewardship of PyPI is a self-fulfilling prophecy: package maintainers realize that the versioning is, essentially, broken in Python, and in order to prevent their packages from breaking, they narrow down the range of acceptable dependency versions as much as possible (and often only accept the single version of the dependency they tested with). This, in turn, results in a situation where upgrades are impossible without massive breakage: incremental upgrades become impossible, because no part of the dependency tree can be upgraded independently. On the other hand, if a single dependency in this tree goes stale, the whole tree is stale.
Another factor to this is that dependencies which accept a wider range of transitive dependencies are harder to install using currently available package managers / installers. I.e. because pip / conda will spend a lot more time trying to find a solution for dependency graph with more potential candidates (due to more loose version requirements), developers are prone to try and expedite the process by narrowing down version constrains. This is especially painful in conda world, where the solver can take many hours (or days in pathological cases) to solve such installations. And so developers have to choose between being more friendly to the user (and supporting multiple transitive dependency versions) or being more hostile to the user (by narrowing down supported versions) while speeding up their deployment.
Yet another factor is the design of Python module system that is unaware of module versions. I.e. if only it was possible to specify the version of the imported module at import time, a lot of problems would've gone away. There would've been no need for virtual environments. Subsequently, there'd be more incentive to ensure compatibility of various independent libraries and programs, which would put pressure on package maintainers to dedicate more time to integration testing and slow down the release cycle.
I've lost count of the number of times runtime behavior and performance issues have been traced back to (for us) far-from-optimal design choices in third-party libraries that probably seemed like reasonable choices to the library authors. Particularly in open source, you see remarkably little investment in performance engineering or scalability, so it is always straightforward to write your own implementation that is many times faster on the same hardware. Even basic things like popular data format conversion libraries are surprisingly consistently suboptimal. In most cases, the kinds of changes we'd want to make cannot be upstreamed to that library; most open source libraries heavily optimize for simplicity and the widest set of users to the exclusion of maximum performance. In an age when hardware can deliver >100 GB/s of throughput, many popular software libraries deliver <1 GB/s per core even though far better can be achieved with a modicum of diligence.
Another issue is the impracticality of threading fine-grained metrics and observability through a hodgepodge of libraries that were never designed to be used that way or to work together. This is a big deal if you want to heavily automate operations.
There is also a real issue with supply chain risk in terms of both quality and security. The vast majority of open source libraries are implemented to a lower standard of quality and security than the rest of the code base. And those libraries are often not really designed to be testable to the extent that would give similar confidence in the implementation quality. Combine this with the reality that the software needs to be deployable in environments that require higher than average assurance makes this a business risk.
As for the manpower cost, it has been low in practice. An engineer-month here, a few engineer-months there, with the benefit of better performance, more optionality, and much easier maintainability. In most cases, we would only use a tiny subset of library functionality regardless, so it adds a large surface area for a narrow benefit. Much of this code is also reusable. The existing open source libraries are very useful here because they provide something to measure against.
I also like that this forces a discussion of actual requirements for that functionality. Far too much software makes their requirements whatever a library offers.
I arrived at this point gradually over many, many years. The practical maintainability and opportunity for optimization have been key. In far too many cases, it turns out that a library dependency could be replaced with a couple hundred lines of code that actually worked better and which matched the design and style of the rest of the code base. There is a learning curve but once you've written a common class of library once, it is straightforward to do it again.
> Particularly in open source, you see remarkably little investment in performance engineering or scalability, so it is always straightforward to write your own implementation that is many times faster on the same hardware. Even basic things like popular data format conversion libraries are surprisingly consistently suboptimal. In most cases, the kinds of changes we'd want to make cannot be upstreamed to that library; most open source libraries heavily optimize for simplicity and the widest set of users to the exclusion of maximum performance.
Yeah, I have even encountered this myself once or twice - i.e. re-implementing some optimization/search algos, sampling processes, or numerical simulation methods to be more tightly coupled to our requirements/context for great performance gain - despite not being in a context where heavy performance engineering has become necessary yet. This obviously overlaps with your point about using only a tiny subset of library functionality.
> Another issue is the impracticality of threading fine-grained metrics and observability through a hodgepodge of libraries that were never designed to be used that way or to work together. This is a big deal if you want to heavily automate operations.
Yes, I personally have not ever run in to this because of the relatively small scales I am operating at but makes perfect sense. Same goes for the security concerns. Obviously these are more serious concerns for larger and longer timescale projects (as discussed in the original blog post).
> engineer-month here, a few engineer-months there
I've only worked in contexts where there are no more than 5 simultaneous active devs on a project, so the long-term investment for longer term gains is probably in higher tension with the short term needs. obvious red flag for increasing technical debt/deferred maintenance/downstream complexity, but still can be difficult to justify diverting 20% of team's total productivity for a long term investment when there are short term demands. But always good to be reminded that those long term investments shouldn't be rejected outright and should be more seriously considered.
> I also like that this forces a discussion of actual requirements for that functionality. Far too much software makes their requirements whatever a library offers. I arrived at this point gradually over many, many years. The practical maintainability and opportunity for optimization have been key. In far too many cases, it turns out that a library dependency could be replaced with a couple hundred lines of code that actually worked better and which matched the design and style of the rest of the code base. There is a learning curve but once you've written a common class of library once, it is straightforward to do it again.
This is a really excellent and concise way of summarizing it. I'm definitely going to try to push this with my team more often and now have a great resource for how to articulate it. Thanks again for taking the time to explain your perspective.
Your home grown auth code doesn’t have the minds and the scrutiny of security experts like the go-to third party auth lib everyone uses does, nor does it have the free pen testing, security fixes, maintenance, optimizations, refactors, ever collaborative maintainers, or corporate sponsorships.
Auth is a solved problem by minds much smarter than my own. I’m glad to outsource things like that to third party deps.
so why maintain your own implementation of the scheme/spec when a community already does, as well as handles the vulns as they pop up and generally has more eyes looking (and fixing) issues?
Here's how I look at it; Trust my own NIH implementation of a spec and hope a user reports a vuln instead of it going undisclosed/unnoticed in the wild for years, or use a community lib that is always being scrutinized and vetted and has security professionals involved. It's a no brainer for me.
Is it? It seems like all the outsourced auth has lots and lots and lots of problems, issues, etc. along with being remarkably expensive.
I'm not saying I, personally, can do better, but it seems like auth is far from a solved problem. Especially since it seems like the difficult part of "auth" is actually "customer support" rather than any technical issue.
(I would argue that the lack of an open-source "Auth in a Box" seems to also argue that auth is far from "solved".)
Spring Security is complex and brittle, I have seen a push to use underlying libraries e.g. nimbus in real-life projects.
It is solved until you have to integrate with third party home grown implementations and providers that implement the specs except for a little bit of behavior that is not in the spec.
And while I kind of agree with you on principle, I've also seen way too many shoddy OSS libraries from "experts" that I had to rewrite to really make it a no-brainer statement.
Just like OP, I'm leaning against shoveling in dependencies if I can help it as I age.
Imagine a doctor making such statement about their field of work. Do you have a degree?
Tell you what. I'll never become a specialist if you never become a doctor. I'd rather you not refuse to admit that there are better doctors for a task than you if my health is in your hands.
I have been trying to get my head around OAuth, zero trust, Fido, SAML et al - and I am not sure I get it.
It’s not that say “JWT tokens might be dangerous” it’s how to hang it all together in a sane manner.
For example I was listening to Security Cryptography Whatever podcast and one of the guests (Chrome dev?) said obviously don’t trust everything that comes down an established TLS connection but verify / auth each request.
That’s a pretty tall order and seems to nullify quite a lot of established frameworks
But realistically I think a lot of frameworks and standards serve different needs than “smallish number high value clients”
This is a great resource to learn how to write it yourself, and things to take into consideration: https://lucia-auth.com/
IMO, this is one thing to implement carefully if need be (vs avoid altogether, as is the case of cryptography).
I also left "auth" ambiguous and didn't specify "authentication" or "authorization" because my original comment applies to both imo.
Probably because auth abstractions are so complicated is why developers started thinking auth is hard when all it was doing was putting a uuidv4 in a session_id cookie.
Like writing SQL without an ORM, developers have been psyop'ed into thinking things are so hard that they shouldn't even try.
Mozilla's Firefox CI builds with multiple compilers, including clang trunk and (IIRC) Rust Nightly compilers to identify code issues that will need to be fixed before Firefox can update to the new compiler versions and to catch regressions in the compilers themselves.
Good Point.
I would love a deeper insight into what you mean by this. I can't imagine you're rolling your own http server, database communication layer (as in like, postgresql's jdbc driver), json parsing engine, etc.
A good case where people use an unnecessary dependency is calling REST (or other http) APIs. If there is a wrapper, people tend to use the wrapper, which is not really needed if the in many/most cases. AT the most use something like requests which is what a lot of things use anyway.
> django-importexport used a savepoint per row making it useless
That is pretty crazy.
It is also another example of something that is easily avoided.
(Even if you need to implement something like that from scratch it may be OK, your requirements don't need all of the functionality a third package includes because someone else needed it)
For example, there are corners of the Perl ecosystem that have been really stable for a long time. Perl itself is very conservative with backwards-incompatible changes, and DBI, the module that abstracts away database interactions, has been around for a long time (the oldest changelog entry is from 1995, but that was for version 0.58, I'd be surprised if it weren't even older).
Of course, then you don't get the fancy, modern libraries, but going with DBI plus DBD::Pg is probably more efficient than coding your own, and stable enough for a ten-year horizon.
Another approach might be to move the really critical pieces of your software to a much simpler interface. For example, HTTP/2 looks much more complicated than HTTP/1.1, so your software might only implement (a subset of) a HTTP/1.1 server, and then you can use an off-the-shelf reverse proxy to expose it to the "modern" internet. This reverse proxy then is easier to replace, because it doesn't handle the core logic of your application.
Or you go really old-school and use CGI, which is an even simpler interface than writing a HTTP server, and any HTTP server that implements CGI would do as a frontend.
So much so that Perl 6 became a whole separate (doomed?) language after trying to change too much.
Some third party dependencies are so widely adopted and supported, they are on-par with the main framework you are using in terms of support and development. They often don't lag the main framework in supporting new versions for long. I don't think these kinds of dependencies have the same drawbacks. You only have a couple at this level anyway, and should choose them carefully.
There are also dependencies that solve one particular problem well and very narrowly couple to a specific feature in your product. If they become problematic over time, it is easy to just replace them or even rewrite it yourself. Specifically, you could select these judging how well you could maintain the sourcecode yourself in such an event. Often, these are easy to extend as the focus is so specific. You can consider just adding it to your codebase and thinking of these dependencies as your code, except that it is written by someone else.
This depends on what kind of dependency you're talking about. For example using Zephyr RTOS significantly reduces time to market and it brings in a lot of 3rd party libraries that are written specifically for it. In other fields this might differ, but fields like Web development are a nightmare to begin with.
I think I understand your sentiment here. For me the risk is that you are not just trusting the original author, you're trusting all future maintainers, especially if you follow your first (and good) suggestion of "aggressively updating tool chains".
The other risk is what you are mitigating by aggressively updating: things break through decay and obsolescence. That can usually be mitigated through updates, but if you introduce a dependency that is eventually abandoned and not given a quality fork then you are suddenly in need of solving this dependency in an entirely new way.
I've adopted a policy that for every significant compiler or build system release, we create a branch that verifies a good build. If it doesn't build without warnings or errors, or fails in testing, that is treated the same as a bug in the code to be dealt with ASAP. Prioritizing this also tends to enforce good build configuration hygiene and automation. Not being able to do this is a bit of an architecture/code smell anyway. This also makes it much easier to incrementally modernize and refactor the code base with new language features -- the tool chain supports the latest and greatest even if the code base doesn't yet.
I also find it difficult to rationalize third-party dependencies these days, they almost all become disappointments over the long-term. The benefits rarely justify the risks. Anecdotally, this policy has not materially reduced development velocity; the scope of useful functionality can usually be replaced with a small amount of code that has good mechanical sympathy with the rest of the code base because it was designed with the code base in mind. I've found in many cases you'll write just as much code trying to interface a third-party library.
That said, for new projects it is often useful to use a few third-party dependencies to plug holes with a clear plan to replace them in the near- to mid-term. Stopgap dependencies have been a reasonable practice in my experience.