On the contrary, I believe we do. The answer is ongoing maintenance.
The basic problem is that people persist in looking at software as a fundamentally mechanistic artifact you finish and ship in a static environment. Then you move on to the next thing. This is fundamentally incorrect. Software engineering is a process that takes place in an adversarial, human-driven environment. There is fundamentally no automating this away because human decision-making work is required.
Any security expert worth their salt will tell you this. As long as you persist in trying to view software as a fixed artifact in a static environment, you will find there is no meaningful answer. Those who come to terms with the dynamic, human-driven reality will find that there's a well-understood - if inconvenient and expensive - answer.
That’s just one example. Medical machines are expensive as hell and companies don’t want to rebuy or constantly invest in something that works and does one thing well.
Software is never just an artifact. It's a whole system, of which the source code and binaries are some components. If you do not manage those artfacts, the next best option is generally managing the environment.
I probably wouldn't bother to try to port it to modern versions of Qt or whatnot even if I hadn't lost the source code 15y ago.
So yes, it happens, it's possible, it sucks
To give you an example: XLSX parsing. For quite a while, the most performant library ‚xlrd‘ had a big scary unmaintained notice on GitHub, yet people kept recommending and using it, often unnoticed through pandas, because they don’t read docs. I‘d refuse to touch that and instead use openpyxl, and now newer xlrd releases have even completely removed xlsx support.
When you choose software to solve a problem, you're not just choosing software. You're encoding expectations around the problem, the environment, and your needs. If all three of those can be assured to never change then choosing a fixed artifact might be the way forward. Should any of those ever change, then your software will either adapt or become a less suitable solution.
How often can an organization really be that certain?
If you need to make guarantees about your product that are dependent on another piece of software, then you just have your vendor produce a spec and guarantee conformance to the spec, then you audit that the spec fulfills your fixed needs and audit that the implementation conforms to the spec.
Then, if you made a mistake, your liability is limited to your mistakes instead of your mistakes and the mistakes of your dependencies.
If, on the other hand, you do not get any guarantees from your dependencies, and you fail to achieve your guarantees then that is entirely on you as you are the one transforming the absence of guarantees into guarantees.
The issue: ML workloads are very sensitive to the exact version of a framework/library used and these frameworks/libraries are not stable despite what their semantic version says.
So what do you do when you have a python vulnerability? Best case, an update "just works" but in many cases you either need to retrain the model (can be extremely expensive or inconvenient), rewrite it if APIs changed, or hunt down a difference in model output if some downstream dependency broke something. Stability of software in the python world is abysmal.
With a service it's all much easier because you can have pretty robust integration tests and you're good to go. With an ML model it's a lot trickier because even if you have sample payloads and responses there might be other payloads you didn't think about that could cause issues. Or it might rear its ugly head when you finally retrain the model and suddenly model performance is crap.
One recent example that comes to mind is that PyTorch now disables Ampere GPUs' TF32 units by default since there were some hard to find edge cases where the matrix multiply results were completely wrong compared to the expected result. This went mostly unnoticed except in a few cases where users were trying something a bit more sensitive to such issues.
I had a similar issue in one of my own models where there was an error in a normalization layer's implementation in Tensorflow (it was computing the average over the wrong axis iirc). It only became noticeable to me when I came back to retrain the model a few months later with an updated Tensorflow version and found that the results from training were not even comparable.
In every other ecosystem when minor version are updated there are very rarely issues, and there are little issues with compatibility with minor versions of libraries.
In part due to a culture that encouraged monkeypatching. With the result that what behavior you get depends on who last monkeypatched it. So a dependency 3 deep that loads the module you expected to load last suddenly causes the monkeypatch you were depending on to disappear...
If you are willing to run other people's arbitrary (Python) code, then - vulnerability or no, you should consider the system running this code as compromised.
This policy evolved, if memory serves, after a security issue was discovered with the XML library we were using. The fix was not back ported to our version, and the versions in between had multiple breaking changes, so it was a slog to fix it.
It wasn't even that we were having a production issue, because we were still in development and internal testing. Not a single 'real' signature had been generated yet. It was all test assets and test CA certs. But we had enough wisdom to put 2 and 2 together and get 4, so we tried to treat the situation as a dress rehearsal for some later bug that happened after we started playing for keeps. Almost everyone could see this was an untenable situation.
There were consequences of course, but we made a bit of lemonade in the process. Our integration tests were expanded to include a larger percentage of pinning tests for our dependencies. Given the gravity of the situation, we needed those anyway. We just now had a poster child for doing the work.
how did that go ? I often wanted to do it but never got to submit the idea.
It also took some reminders from the leads to add it in instead of letting it slip, but so far it's the least broken process I've gotten to use.
Sorry, this is a total strawman. Automation doesn't need to be integrated with the build in such a way that it creates noise, and it doesn't mean you need to let changes be applied without human intervention.
I'm hooked up with Snyk which scans all of my dependencies and sends me a summary email if it discovers any vulnerabilities. It creates PRs that can patch dependency vulnerabilities, and I can apply them if I approve of the change. If I don't think the change is worth it or the vulnerability is relevant, I can go to the Snyk site and mark a vulnerability to be ignored, or even tell it to ignore it for a month or whatever. I'm sure Snyk isn't unique, either.
Yes, a human should be involved in decisions. No, it doesn't need to be noisy.
I'm on Node.js and working on multiple projects with crazy numbers of dependencies, and the number of "new vulnerabilities" I see is typically a few per month at most. Some months there are no new notifications at all. It simply doesn't change that quickly, and it's not constant noise unless you ignore the notifications, and that's just bad software engineering.
I am currently fighting with our security team because they refuse to allow developers to do this, and instead require my team to file tickets with their team, wait hours for that ticket to be picked up, explain why a thing is a false positive to a person who doesn't understand the codebase they're working on, and hope that person feels that the finding should be ignored.
I do not work at a large company.
This is greatly exacerbated by the fact that Snyk - like all package scanners - primarily flags false positives. When you start digging in to what it flags you'll find that a lot of the vulnerabilities aren't reasonably exploitable, or the impact is questionable, or you can't repro the issue. Those that do legitimately affect the underlying library often don't affect your usage of that library. I'd estimate our false positive rate is >90%. So when we get a finding our options are to go through that costly and disruptive process I mentioned above, or just upgrade the dependency (which carries associated risk of breaking shit in production).
These tools emphasize the quantity of findings they produce, and quietly ignore the quality. As soon as you have any divergence in motivations or incentives between the people running the scanner and the people being blocked by the scanner those low quality findings become incredibly costly.
I'm sorry you're on the receiving end of this problem!
Shill notice: I'm working on an Open Source tool[0] that makes this problem less horrible. My colleague wrote a post about our hypothesis[1] about how we can avoid this false positive trap.
I'd love to chat with anybody feeling this pain (even just as therapy lol).
0: https://github.com/lunasec-io/lunasec
1: https://www.lunasec.io/docs/blog/the-issue-with-vuln-scanner...
The Snyk tools apply to the patch directly to (e.g.) `node_modules` after the latest code has been downloaded.
At this point, I think a lot of problems in software are actually misaligned or misincentivized policy gone awry. When you read about them on forums they look technical, but they're not.
And then there is the "I want the fix, but I don't want the added attack vector of features that comes with it."
Honestly it's just not worth worrying about whether a security hole is exploitable when the decision is whether to accept a minor update patch. Just accept the patch PR and move on.
The times you need to do a major update to get a patch fix it's equally important to just make that upgrade, since that means the ancient version you're using is likely out of maintenance and likely is throwing warnings that you're using a deprecated version.
Yes, it's more work to perform a major upgrade. But that doesn't mean it isn't good engineering to keep your tools up to date.
That is, I think the ask is more of "why haven't we come up with something that is a bit better at mitigating risks, here?"
I believe this is one of them.
To add further insult to injury, colleges send SWE/SDEs to the work force to learn to 'actually code', while companies have no time to train jr developers so they are just thrown in the fire and hope for the best.
1. keeping it running with a healthy userbase and demonstrating high impact does, and
2. if you know how to spin it, then maintenance work can look like "building new stuff".
You've worked for some great orgs, then! My experience is mostly the opposite. One company I worked for did value mentorship, at least somewhat, and it was a factor in promotion decisions, but not mentoring others didn't really stop people from getting promoted.
1) We use very few dependencies. We err on the side of writing our own function, library, or UI component rather than installing a new dependency, even if a "better" open-source version might theoretically be available.
2) We use a monorepo. We have two Go backends sharing the same go.mod, and three Typescript/React frontends sharing the same package.json, all in the same repo. An upgrade for one is an upgrade for all.
3) Every first Friday, I update all dependencies to the latest version. Including major versions. Yes this is a pain sometimes (like when I realized we were on Webpack 4 and the latest was Webpack 5), but if you do it consistently, it's usually not that much trouble. 1-2 hours per month on average.
In terms of security, I think this is a good middle ground. The approach has other benefits than security, too.
This just means you likely have a lower quality version with the same problems but without the benefit of an ecosystem to find them.
We won't write it ourselves if we aren't sure we can do it well, i.e. we're not going to roll our own encryption. Those are the types of dependencies we do use, and upgrade once a month like clockwork.
The point is not "don't use someone else's encryption library," the point is "don't use someone else's LeftPad() function that takes 3 lines to write". That gives us fewer dependencies, which makes it easier to upgrade all dependencies regularly, which is good for security.
While this may solve the "exploit from 3rd party dependencies" this says nothing about the security quality of your functions. Now instead of the vuln being found in the 3rd party package and fixed, the issue remains in your program forever, probably getting exploited by nation state level actors without your knowledge.
Your boring business systems that will live for 50 years need to have boring system level libraries or stuff you maintain. Your fast time to market or non-critical systems are optimized by speed and cost.
If your service depends on a bunch of downstream stuff, your processes need to support upgrade cycles. If you can’t keep up with that, you need to refactor.
Some of this is trivial, and a very good thing - stop using bullshit exercises in fragmentation like isEven.
Some of it is very much not, and will apply pressure to find vendors who are willing to vet their own bundles of commonly used libs.
There will be numerous second order effects of that, including the ghettoization of open source that isn't "QualStrike Approved(tm)".
Capability-based security is one possible answer. The Austral language is trying to make this a first-class language feature:
>The problem is that code is overwhelmingly permissionless. Or, rather: all code has uniform root permissions. The size of today’s software ecosystems has introduced a new category of security vulnerability: the supply chain attack. An attacker adds malware to an innocent library used transitively by millions. It is downloaded and run, with the user’s permissions, on the computers of hundreds of thousands of programmers, and afterwards, on application servers.
>The solution is capability-based security. Code should be permissioned. To access the console, or the filesystem, or the network, libraries should require the capability to do so. Then it is evident, from function signatures, what each library is able to do, and what level of auditing is required.
1. Depending upon context, *any* change to state (network I/O, console I/O, file I/O, environment variables, etc.) is a security vulnerability you can drive a truck through. For example: Foo is a very-locked down package that only appends text to a file the caller owns. It's formally verified to only change the file specified, verified to only append in all possible situtations, verified to not have cross-file-system identity problems, not subject to file renaming race conditions, yada yada yada. Safe? No. "foo('evil_command' '~/.bash.rc')".
2. A useful definition of a "capability" depends upon context. A *dynamic* context. Say Foo is intended to be used for generating log files. There's no way for a third-party library to determine whether the target is actually such a thing. It requires manual tuning which fits the operational context. Perhaps the file must be named "/var/log/${x}.log". Or maybe its "${home}/local/log/${x}.log". Great. Of course, now you've got to verify the context-specific defintions. And verify the tool that does this verification. And the verify the configuration of the tool that does the verification... Safe now? No. Context is *dynamic*. "foo('var/log/evil-hack-attempts.log' '0.0.0.0/0 tried-to-crash-us')", and your other context, the automated blacklisting, locks out the Internet.
3. This gets very complex very quickly. IMHO, configuring Selinux, in a real production environment with real personell, is akin to rolling your own cryptography. Can people learn to do it? Of course. But the same argument applies to writing cryptographic functions. You. Will. Make. Huge. Mistakes.
To be clear, this does not mean that capabilities are useless. It's obviously critical that they exist. However, they ain't magic. You can't "solve security with capabilities." There is a point where throwing more of them at the problem makes things *worse*.
for example we didnt fix how we build applocations and manage dependancies, we invented docker.
we didn't create a secure runtime for the cloud where applications can run, instead we virtualised whole operating systems
"we do not break userland, period" -- Linus Torvalds
not breaking existing stuff is why many abstractions exist, and even why many are put in place early on, to allow for some wiggle room without having to break all the things. people tend to prefer software that continues to work, above all else.
Abstractions that paper over problems that you are unwilling or undisciplined enough to tackle properly are usually bad. I think the grandparent was talking about this kind.
some abstractions carry decades of legacy (e.g. linux file-system hierarchy), and cannot simply be replaced. things that did not connect to the non-existent internet did not have to paper over problems that simply could not exist at that time (VMs, multi-user, multi-tenant hardware). there is plenty of deep software dependencies that are no longer maintained which still rely on these abstractions and cannot be cheaply/reasonably replaced in the field. sure, we have new abstractions since then, but only new and actively maintained software can ever make use of them, and that will only end up in systems that are also actively maintained instead of simply continuing to work with exactly zero effort.
The whole log4j thing happened is because java is very poorly designed from the get go, and one of those points is the insistence of making everything an object, including data. And of course, you often need to output the string representation of data.
From a theory perspective, if you want to rely on language features like strong typing to ensure correctness, then you have to take it all the way to full provability, where you explicitly guarantee bounded input, bounded output, and any side effect (where a network call is a side effect, and if your code doesn't explicitly state that this side effect will happen, it won't run).
Not sure if it's the best, but my current approach is to target major revisions/versions with ABI/API compatibility as viable candidates, and update always, but at a lagged rate, ideally allowing enough time to pass for vulnerabilities to be surfaced in those versions. Log4j is a tough case due to how far back it goes version-wise, but you simply can't win them all.
Another important way to minimize risk is to have a strong ability to provide a corrective action. If replacing the version or the entire dependency of a library you use takes a significant engineering effort, then you are arguably at a higher risk due to that than you are in your decision to never vs. always update.
The only complete solution would be static analysis (read: type systems) that can guarantee everything about some code's contract/behavior relative to the caller. Short of that, it's just going to continue being a bunch of manual half-solutions like API checks, existing type-level checks, manual changelog reading, manual upgrades, testing, etc.
That's my armchair take on it.
Aren't we already doing that, to some degree, by watching for notifications of vulnerabilities, and then manually patching or updating? Sure, with this method, we have the ability to decide to ignore a vulnerability, but I'd guess most people aren't great at assessing risk (and may not fully understand the severity of a vuln, or if their use of the software makes them exploitable), and should probably just update whenever a fix for a vulnerability comes out. So you might as well automate this.
> So the exact software you compile will depend on whether or not you can connect to a 3d party service?
Anyone who builds against public package registries (Maven Central, npm, pypi, crates.io, etc.) already has this problem. Some people will go to the effort of putting their own proxy cache in front of these registries to insulate themselves from downtime, and I expect these sorts of people would do the same for a 3rd-party vulnerability updater service.
Regardless, I wouldn't expect this to be a "pull" model, wherein you wait until the next time you'd build before getting security updates. More likely you would get a notification or even an automatically generated pull request (like GitHub's dependabot does) when a security update is available. Then it's up to you whether you want something else to automatically apply those updates, or you can review them yourself and apply manually.
I think you're just making a mountain out of a molehill. There are several existing options to automate or partially automate this, depending on your trust of the relevant third party. And you can still do it manually, if you so desire.
You could formally spec software and implement it, congrats you're now 1000x slower to market and any change you want to make is another 1000x investment to spec out and prove.
And let's not even mention the ridiculous idea that open source developers be expected to do this. Moreover, this doesn't even handle hardware problems!
Okay, but we could just build all the software internally, reinvent all the wheels. But how is this any better? Now you have even less manpower to fix bugs and you're probably 100x slower to market because you're building all these half baked, bug ridden libraries.
If anything, the best idea is formally verified sandboxing, so that you have strong assurances about certain apps not doing bad stuff, but this doesn't solve all problems either.
It's an unsolvable problem in general, which is why no one has solved it.
I think until Software Engineering catches up in this regard with traditional Engineering disciplines we'll continue to see issues arise from the software realm. As much as the self-titled "Engineers" proclaim it, there's more to engineering than design and production.
I've been asked why I got my P.Eng as it does nothing for me career wise, but I always have the hope that we move towards a profession that does utilize engineers with proper accreditation and responsibilities where appropriate. I work in Industrial Controls and what my job has me doing does have safety concerns. We fired a student at one point working for us because his attitude towards the work was lackluster and lazy. I see it often enough in programs I'm asked to pick up and fix too.
If Software Engineering wants respect, the profession needs to accept that we're going to piss off a lot of Developers & Architects using the title incorrectly. We still need to develop the proper rigor to take Engineering as a discipline seriously, but we also don't need to apply "Engineers" to every job.
You’ve brought up three relevant points: 1. The tools are software, which strength and weakness is being easy to change. 2. The tools rely on abstraction, which strength and weakness is summarizing/hiding complexity. 3. Communication between publishers/consumers of these tools is hard.
Armchair-coaching I now come up with at least three solutions: 1. Re the weakness of #3: ensure when an issue is found at one layer, communication happens to maintainers of all other layers up and down. 2. Re the weakness of #2: collapse the layers – analyze dependencies and don’t go past more than x number of abstractions. 3. Re the weakness of #1: don’t use software.
If you using a library you have to spend time on keeping it up to date (which includes testing and fixing whatever when wrong during an update).
Or even worse you may have a legacy "fat jar" that you can't even manage the transitive dependencies on - they're compiled in, and that's that. Something else has a version conflict? Tough shit, one or the other is going to break, and depending on how bad your build is, you may not even get an explicit choice, it may be thrust upon you by the classloader.
Sometimes you can mitigate it somewhat by encapsulating them in their own service and just firewalling the hell out of it, and modules should help the dependency conflicts somewhat, but the problem is these are basically "UXO in the backyard garden", they are a passive business risk and sooner or later you may get lucky and they go boom even though you did nothing wrong.
Despite your best intentions, sometimes you gotta update dependencies, and if you allow it to fester, then you risk that the time you have to solve the problem is during an urgent security crisis. Generally it is better to update them on your own timetable than to have it forced upon you like that. And the problems and limitations and workarounds only become worse and worse over time if you choose to actively ignore it.
It'd be cool if devs didn't suck ass at writing code but that's the world we live in.
If you don't update your dependencies all the time, you will be vulnerable to old bugs and issues."
>has no meaningful answer to this...a sane industry this realization would lead to a change of the paradigm
This sounds like gene mutation and evolutionary selection pressure. It's sort of a fundamental aspect of the universe no? There may not really be a meaningful "answer" to this in principle just various half-measures to muddle along and adapt to this reality, survival of the fittest in a sense.
Why the scare quotes?
>In a sane industry this realization would lead to a change of the paradigm, but people in our industry seem to be only doubling down.
Maybe because half the time people with extensive training/experience in security start to even say something they are completely disregarded (like here), or are hamstrung by budget because security is viewed as a garbage disposal that profits get shoveled into, or because they are hamstrung by pushback over every little change because people seem to think the main goal of someone doing security is pissing off users.
I'm not making fun of you or trying to be funny - it is simply an observation that I find interesting.
Also nobody that knows what they're doing uses "duct tape" on ducts, so the name is pretty meaningless.
This is already true with the vast majority of software using any an online software repository, e.g. Go, NodeJS, Java, Rust, etc.
With security, you need some online service so you can find the new CVEs every day.
Or you just do it manually if that service is down. It's weird to me that the argument for not having an automated, 3rd-party service is "if it goes down then you'll have to do things manually", when the alternative is "you always have to do it manually".
If you are comfortable trusting a third-party service to tell you when to upgrade, then that is absolutely an improvement over doing security updates manually. This is why I have unattended-upgrades set up on my Debian systems to automatically install updates from Debian Security every day. Sure, it may fail for whatever reason, but I am certainly not going to take the time to (or even remember to) update every day.
But basically if you just subscribe to a few projects' releases you can pretty easily get things pushed to you when it matters.
Also known as doing your job.
I am no expert. I ended up indexing all the open source kruft I use to hold this ship of fools together, then verified that the Log4J pieces were definitely disabled with a bunch of monitoring while I tossed stuff at it. I did mention I am not an expert, right? I am sure there is a more Pro way to do this.
solution: stop eating
Option 2 - which you are advocating - it's every man for himself and we decent into socity of warring tribes of hunter hatherer or subsitence farming at best
let the buyer be aware
But, you're not wrong, some people reach for every library and framework they can, and others focus on building the most minimal and understandable thing that will work. I'm not saying writing your own server in assembly, but, avoid unnecessary junk and magic and libraries.
It's the classic dilemma: in this world of ours, one can either return to monke or progress to crab.
The answer is to stop using garbage software written in garbage languages. Of course the current environment is too high-time-preference for this to be economical in many industries.
We have lightweight-to-heavyweight formal systems which can almost completely or literally completely eliminate issues of the type seen with log4j, but due to the marginally higher up-front cost of using such systems, only firms with some combination of high stakes, low time preference, and foresight end up using them.
I assume it's one with GC.
well, back to the mainframe it is then
It isn't a language problem, it is a developer practices problem. Developers code like security isn't a thing then are confused when their programs aren't secure.
I'd prefer that the "fix" for this isn't "require that millions of developers around the world all change their behavior, and never screw up in the future".
Languages can still protect against these sorts of bugs. Effect systems can describe as a part of the type system or function signature what kinds of things a function can do. For example, you could have a logging function where you know that the only side-effect is that it can write to stdout or stderr, or a file. Running arbitrary commands would not be allowed, and would cause the program to fail to compile.
As a real-world example, I've used the Slick SQL library in Scala, and the query type has a type parameter for whether the query is an read or a write. It's easy to build a simple abstraction on top of your data layer to never allow a write to go through, for example, if that's something that's useful to prevent bugs. I think one thing I did with that was set things up so callers could not accidentally write to MySQL replicas; only reads could go to them, and writes had to go to the primaries.
Granted, it takes discipline to build and use such a system, especially one that has such fine-grained effects. (I would expect most are more like a binary "does not do I/O" or "does I/O".)
It isn't a language problem
Exactly. This flamy sub-thread has no point here.
If you don't update your dependencies all the time, you will be vulnerable to old bugs and issues.
The current software engineering paradigm has no meaningful answer to this, no matter what "security experts" tell you. In a sane industry this realization would lead to a change of the paradigm, but people in our industry seem to be only doubling down.
"Use a build tool to do dependency checks!"
And then what?
"Only update libraries if they have vulnerabilities."
So, should this be a manual process where I have to dig through obscure warnings every time I build something?
"No, you can automate it!"
Congrats, you've just outsourced the decision making process about which versions of libraries to use to some 3d party. You've also made your build process reliant on yet another online service.
"But it's not a hard dependency, you can still build if the 3d party service is off."
So the exact software you compile will depend on whether or not you can connect to a 3d party service? Do you understand the actual implications of this?
Etc. People with clever advice don't seem to think it through. We're fighting fragile complexity of too many tools ducktaped together by ducktaping more tools to the whole setup. Again and again and again.