> The net effect of it seems to be that a full system for really acceptable programming will be at the same time a full system that will suffice for the description of constructive mathematics.
He said that in '85!? Wow.
[1] https://maniagnosis.crsr.net/2007/10/1985-dijkstra-interview...
That's basically the Curry-Howard correspondence[0], which was first explicitly stated in 1980.
0: https://en.wikipedia.org/wiki/Curry-Howard_correspondence
"But why would you do that?" and "But thats the point of using those libraries!" right?
when you're the only one that can find the heisenbug and fix it; your effort and the need for it aren't evidence of over-reliance on dependencies. They are an argument for re-implementation from scratch in brand new, clean, safe technologies!
you may have started a howl
This is seldom actually done. More frequently, customers will find bugs, and fix or report them, thus, improving the tool for all users.
In this one case, it was not reasonable for me to do this. The application that I'm developing is closed-source (for now), and I was in the middle of a pretty massive refactoring job. There was no way that I was going to take a couple of days to try to nail down a random memory leak/thread safety bug in someone else's code, when I could simply use the Apple OS tool to do something similar, and I knew it would be safe.
I definitely could have figured it out (given time). I'm a very good debugger, but it would have been an unacceptable branch in my workflow. I have a pretty full dance card, and this was not on it. Thread safety issues can be a pain to track down, and they are like cockroaches; for each one you see, there are a dozen more, behind the drywall.
I wasn't about to raise a ruckus about it. I suspect that the tool works fine in standard main thread apps, and my application uses worker threads that originate from things like network callbacks. The author won't shut down the tool, because one random person with a specific workflow has an issue. I have managed open-source projects, and know what it's like to be on the other end of that crap.
The biggest issue with dependencies is trust. That's why walled gardens are attractive. There's a chain of responsibility, and some kind of accountability baked into them. Even then, some of these libraries can get compromised, or sold off to nefarious actors. Also, I suspect that many of these SDKs and dependencies are really "first hit is free." They can make us dependent upon a person, company, or whatever, that does not have the best interests of our user at heart.
I tend to rely on manufacturer API/toolsets, and even then, I'm really careful. Some of the SDKs that ship with devices and services can be...questionable. But if I find myself blaming the compiler for an issue, I can be reasonably sure that the actual problem is mine. If I am using a blackbox tool that sits between me and the iron, it can be difficult to figure out where the problem lies.
Just a couple of days ago, I had to change the hosting provider that runs the backend for the application that I'm developing. I wrote that backend, and there was definitely some kind of bug, but the hosting provider decided to introduce a new "can't be switched off" inline cache that completely broke interactions with the backend, so I couldn't find my bug. If the backend had been a server that I don't control, it could have taken many more days to even figure out that the bug was in the host, report it (or diagnose it), and then wait for the inevitable back-and-forth before it was addressed. Since my primary work is the frontend app, that would have meant a huge delay. I figured out the cache issue, because I am intimately familiar with the backend, and could easily diagnose problems, using the classic "divide and conquer" methodology. Once the hosting provider was switched, it took me half an hour to find and fix the bug.
Even that assumption is being eroded
Abstractions are leaky. You should strive to understand where your black box is transparent.
I agree, I've never gone as far as reviewing/understanding source code for all the libraries I use. Might be nice to have the time ;-)
>it is _good_ to black box things
I trust that you are talking about using other people's black boxes, as opposed to reinventing your own?
I would say that there is some element of risk in trusting a library that someone else made, particularly over time as systems need to be upgraded. Will there be problems during or after an upgrade? Is the whole dependency chain upgradable? Will the upgrades work on all operating systems in use? Are there relevant corporate policies and/or whitelists that might interfere with or slow down upgrades?
Personally I pay close attention to the dependencies I'm using in a project, especially long-lived projects. I find it super aggravating when something I've tested and deployed breaks a couple years later after routine upgrades due to some kind of drift problem to do with a dependency.
How much benefit am I getting by using the library? Best is using base library that comes with the language. Next best would be a well maintained third-party library with a good reputation in the community, one that doesn't have too many dependencies itself. Am I happy with the api of the library, and its documentation? Is is reasonably easy to approach given its functionality? Is the library available as a package in the operating systems I'm working with? How long has it existed? How stable has it been over time? Is the library a good fit for what I am doing, or do I have to bend my code to use it as intended? Do I have multiple libraries to choose from? Can I use the library in such a way that I can change my mind later?
I think there is value in trying to minimize the use of external libraries, up to a point.
It's not about having the time, but -making- the time. For the most part we simply do not prioritize looking into the source code of the libraries we use, to any meaningful degree. (I'm guilty of this for sure). Hence things like leftpad and the many other instances of dependency controversies.
In an ideal world we [and our employers] would put emphasis on inspecting the huge amounts of code we pull in and allocate time for it.
(Cost to dependencies doesn't have to be just review time. E.g. in my embedded software projects, the calculus for "will we need to update this" looks very differently than it would for your typical web app. And extracting interesting bits from libraries and including only that into the main project is maybe more common, reducing the surface of a dependency - which would be a bad idea if you expect frequent changes, but these projects typically don't)
I don't mind dependencies, but I won't use them for anything mission-critical, unless I spend a great deal of time vetting the dependency. That might mean auditing the code, but more often, it is auditing the coder.
The main thing is, is that we need to really think about anything we add to our precious project, from outside. In my experience, people seem to be alarmingly casual about this.
Good results can be building very big systems, in a very short period of time, using very few resources, and everything works great.
Bad results can be building very big systems, in a very short period of time, using very few resources, and everything works great.
Until it doesn't.
For example, an upgrade introduces things like renamed CSS hooks, or introduces thread safety issues. A PR isn't vetted properly, and license-problematic code, or malicious code is introduced into the system. Maybe the author has a car accident, and can't manage the library anymore. Maybe a dependency down the chain gets taken over by someone that uses...let's say "solarwinds123" as the password for their CI server, and they get pwn3d, but, since they are buried three levels down, no one in the main dependency realizes what happened, and so on. Maybe you get to a point where you outgrow your backend, and need to change to a more robust or secure backend, but the SDK you are using has you addicted to the initial backend that you selected, when the project was still in the crib, etc.
It should not be a casual decision.
In my original post, I talk about having to yank a dependency.
It was a phone-number parser. If you know anything about phone numbers, parsing them is non-trivial. Apple has a built-in utility (a scanner for URLs, phone numbers, addresses, etc.), but that is not as effective as the parser I imported, which had a fairly good pedigree. I looked at the code, and checked out the author, and it looked good.
The issue that I encountered was most likely (can't be sure) a thread-safety issue. I ran parses in non-main threads, and noticed memory leaks, and random crashes.
I do not suffer memory leaks or crashes of any kind, in my apps, so it had to go.
Phone number parsing wasn't a dealbreaker. Because it was a dependency, its application was fairly encapsulated (I always encapsulate external dependencies -just plain old horse sense), so I can go back and maybe reinstate it (if I can fix the issue, or use it only in main threads), or add a different one in the future. In the meantime, the Apple tool is fine for my development.
The backend for the project, on the other hand, is mission critical, so I wrote it myself. I am not a backend expert, but there are some things that I needed assurance on, that no backend engine can give me. It's highly encapsulated and layered, so future efforts can replace it. I can tell you that whatever it gets replaced with, will need to meet my exacting criteria.
My current project has one external dependency (it has quite a few, that I wrote): A keychain abstraction. It's very simple, and is actively maintained. I've used it in other projects for years.
The dependencies that I wrote are pretty intense; even the small ones. The testing code dwarfs the implementation code in all of them. One of them is an SDK for a pretty vast system, where I was the original architect, but has since passed on to a new team that has my complete confidence; so I guess you could also call that an "external" dependency, but one that I know very, very well.
This is a very unsafe assumption, at least until we invent a programming language that disallows bugs of any class.
I do! In fact if we exclude the language implementation itself[0], a plurality of my projects have NaN% of their dependencies fully audited.
0: ie, compiler, hardware, etc; if you object to this, please do explain how to eliminate these dependencies; I'm quite interested.
It's a choice to be ignorant of your dependencies, not a law of nature.
The fact that most people don't audit code they depend upon is a problem to be fixed, unless the underlying code is fixed first.
... to teams you can 100% trust and whom are in the same sphere of responsibility. So yes, take a binary library blindly from the team down the hall who reports to the same VP.
From an unknown rando on the internet, not so much.
Unfortunately, this is pretty much the only way, besides the "walled garden" approach.
I am often derided as being a "grumpy old control freak." They def have a point, but I am very, very leery of bringing code into my projects that I don't understand, and stepping through source takes almost as long as writing from scratch.
Most dependencies cover a wider range of functionality than the narrow application that we seek in our individual projects, so, to truly understand the dependency, we need to examine all of its faces.
I recently had to yank a dependency out of a project that I'm developing, because it had a regex bug that caused memory leaks and random crashes in worker threads. It would have been way too difficult for me to figure out what was happening, so I removed it, and wrote my own functionality that is not as good, but doesn't crash my application. This is the kind of behavior that gets me scorn. I'm supposed to keep the crash, and shrug.
So, yeah. I guess I am a grumpy old control freak...