People in this thread are worried that they are significantly vulnerable if they don't update right away. However, this is mostly not an issue in practice. A lot of software doesn't have continuous deployment, but instead has customer-side deployment of new releases, which follow a slower rhythm of several weeks or months, barring emergencies. They are fine. Most vulnerabilities that aren't supply-chain attacks are only exploitable under special circumstances anyway. The thing to do is to monitor your dependencies and their published vulnerabilities, and for critical vulnerabilities to assess whether your product is affect by it. Only then do you need to update that specific dependency right away.
> for critical vulnerabilities to assess whether your product is affect by it. Only then do you need to update that specific dependency right away.
This is indeed what's missing from the ecosystem at large. People seem to be under the impression that if a new release of software/library/OS/application is released, you need to move to it today. They don't seem to actually look through the changes, only doing that if anything breaks, and then proceed to upgrade because "why not" or "it'll only get harder in the future", neither which feel like solid choices considering the trade-offs.
While we've seen to already have known that it introduces massive churn and unneeded work, it seems like we're waking up to the realization that it is a security tradeoff as well, to stay at the edge of version numbers. Sadly, not enough tooling seems to take this into account (yet?).
At my last job, we only updated dependencies when there was a compelling reason. It was awful.
What would happen from time to time was that an important reason did come up, but the team was now many releases behind. Whoever was unlucky enough to sign up for the project that needed the updated dependency now had to do all those updates of the dependency, including figuring out how they affected a bunch of software that they weren't otherwise going to work on. (e.g., for one code path, I need a bugfix that was shipped three years ago, but pulling that into my component affects many other code paths.) They now had to go figure out what would break, figure out how to test it, etc. Besides being awful for them, it creates bad incentives (don't sign up for those projects; put in hacks to avoid having to do the update), and it's also just plain bad for the business because it means almost any project, however simple it seems, might wind up running into this pit.
I now think of it this way: either you're on the dependency's release train or you jump off. If you're on the train, you may as well stay pretty up to date. It doesn't need to be every release the minute it comes out, but nor should it be "I'll skip months of work and several major releases until something important comes out". So if you decline to update to a particular release, you've got to ask: am I jumping off forever, or am I just deferring work? If you think you're just deferring the decision until you know if there's a release worth updating to, you're really rolling the dice.
(edit: The above experience was in Node.js. Every change in a dynamically typed language introduces a lot of risk. I'm now on a team that uses Rust, where knowing that the program compiles and passes all tests gives us a lot of confidence in the update. So although there's a lot of noise with regular dependency updates, it's not actually that much work.)
I fought off the local imposition of Dependabot by executive fiat about a year ago by pointing out that it maximizes vulnerabilities to supply chain attacks if blindly followed or used as a metric excessively stupidly. Maximizing vulnerabilities was not the goal, after all. You do not want to harass teams with the fact that DeeplyNestedDepen just went from 1.1.54-rc2 to 1.1.54-rc3 because the worst case is that they upgrade just to shut the bot up.
I think I wouldn't object to "Dependabot on a 2-week delay" as something that at least flags. However working in Go more than anything else it was often the case even so that dependency alerts were just an annoyance if they aren't tied to a security issue or something. Dynamic languages and static languages do not have the same risk profiles at all. The idea that some people have that all dependencies are super vital to update all the time and the casual expectation of a constant stream of vital security updates is not a general characteristic of programming, it is a specific characteristic not just of certain languages but arguably the community attached to those languages.
(What we really need is capabilities, even at a very gross level, so we can all notice that the supposed vector math library suddenly at version 1.43.2 wants to add network access, disk reading, command execution, and cryptography to the set of things it wants to do, which would raise all sorts of eyebrows immediately, even perhaps in an automated fashion. But that's a separate discussion.)
Thats because the security industry has been captured by useless middle manager types who can see that "one dependency has a critical vulnerability", but could never in their life scrounge together the clue to analyze the impact of that vulnerability correctly. All they know is the checklist fails, and the checklist can not fail.
(Literally at one place we built a SPA frontend that was embedded in the device firmware as a static bundle, served to the client and would then talk to a small API server. And because these NodeJS types liked to have libraries reused for server and frontend, we would get endless "vulnerability reports" - but all of this stuff only ever ran in the clients browser!)
I once almost managed to get back on the release train. I was tasked with adding new features to a piece of software originally developed years ago for internal use at a university. It was running PHP 5 on Debian Jessie. The first hurdle was that no Docker image existed for the production environment (PHP 5). After patching all the code and getting it to run on PHP 8, another issue surfaced: the MSSQL server it needed to communicate with only supported TLS 1.0, which had been removed from the OpenSSL version included in Debian 12—the base image for my PHP 8 setup.
In the end, I decided to implement a lightweight PHP 5 relay to translate SQL requests so the MSSQL server could still be accessed. The university stuffs were quite satisfied with my work. But I really felt guilty for the next guy who will touch this setup. So I still didn’t quite make it back onto the release train… as that PHP 5 relay counts.
Jumping straight to the new release because it fixed one security bug has always struck me as a round about way of trying to achieve security through obscurity, especially when the releases include tons of other changes. Yes, this release fixed CVE-123, but how many new ones were added?
> Sadly, not enough tooling seems to take this into account
Most tooling (e.g. Dependabot) allows you to set an interval between version checks. What more could be done on that front exactly? Devs can already choose to check less frequently.
>The thing to do is to monitor your dependencies and their published vulnerabilities, and for critical vulnerabilities to assess whether your product is affect by it. Only then do you need to update that specific dependency right away.
The practical problem with this is that many large organizations have a security/infosec team that mandates a "zero CVE" posture for all software.
Where I work, if our infosec team's scanner detect a critical vulnerability in any software we use, we have 7 days to update it. If we miss that window we're "out of compliance" which triggers a whole process that no one wants to deal with.
The path of least resistance is to update everything as soon as updates are available. Consequences be damned.
You view this as a burden, but (at least if you operate in the EU) I’d argue you’re actually looking at a competitive advantage that hasn't cashed out yet.
Come 2027-12, the Cyber Resilience Act enters full enforcement. The CRA mandates a "duty of care" for the product's lifecycle, meaning if a company blindly updates a dependency to clear a dashboard and ships a supply-chain compromise, they are on the hook for fines up to €15M or 2.5% of global turnover.
At that point, yes, there is a sense in which the blind update strategy you described becomes a legal liability. But don't miss the forest for the trees, here. Most software companies are doing zero vetting whatsoever. They're staring at the comet tail of an oncoming mass extinction event. The fact that you are already thinking in terms of "assess impact" vs. "blindly patch" already puts your workplace significantly ahead of the market.
I really dislike that approach. We're by now evaluating high-severity CVEs ASAP in a group to figure out if we are affected, and if mitigations apply. Then there is the choice of crash-patching and/or mitigating in parallel, updating fast, or just prioritizing that update more.
We had like 1 or 2 crash-patches in the past - Log4Shell was one of them, and blocking an API no matter what in a component was another one.
In a lot of other cases, you could easily wait a week or two for directly customer facing things.
"The thing to do is to monitor your dependencies and their published vulnerabilities, and for critical vulnerabilities to assess whether your product is affect by it."
Yes
"Only then do you need to update that specific dependency right away."
Big no. If you do that it is guaranteed one day you miss a vulnerability that hurts you.
To frame it differently: What you propose sounds good in theory but in practice the effort to evaluate vulnerabilities against your product will be higher than the effort to update plus
taking appropriate measures against supply chain attacks.
> People in this thread are worried that they are significantly vulnerable if they don't update right away
Most of them assume what if they are working on some public accessible website then 99% of the people and orgs in the world are running nothing but some public accessible website.
And it needs to be said that you generally cannot tell if a vulnerability is critical for a given application except by evaluating the vulnerability in the context of said application. One that I've seen is some critical DoS vulnerability due to a poorly crafted regex. That sort of vulnerability is only relevant if you are passing untrusted input to that regex.
A million times this. You update a dependency when there are bug fixes or features that you need (and this includes patching vulnerabilities!). Those situations are rare. Otherwise you're just introducing risk into your system - and not that you're going to be caught in some dragnet supply chain attack, but that some dependency broke something you relied on by accident.
I'd rather have excellent quality test suites, and pipelines I can have a high degree of confidence in to catch these issues and mitigate the risk, than dependencies that basically never get updated and then become very difficult to do so, which just introduce different classes of risk.
Vulnerabilities are not rare, the two most popular programming languages, javascript (node) and python, have 1000+ CVEs in their official docker images. I.e. in practice useless and shouldn't be used by anyone for anything.
Also, if you are updating "right away" it is presumably because of some specific vulnerability (or set of them). But if you're in an "update right now" mode you have the most eyes on the source code in question at that point in time, and it's probably a relatively small patch for the targeted problem. Such a patch is the absolute worst time for an attacker to try to sneak anything in to a release, the exact and complete opposite of the conditions they are looking for.
Nobody is proposing a system that utterly and completely locks you out of all updates if they haven't aged enough. There is always going to be an override switch.
The Debian stable model of having a distro handle common dependencies with a full system upgrade every few years looks more and more sane as years pass.
It's a shame some ecosystems move waaay too fast, or don't have a good story for having distro-specific packages. For example, I don't think there are Node.js libraries packaged for Debian that allow you to install them from apt and use it in projects. I might be wrong.
An eco system moving too quickly, when it isn't being fundamentally changed, isn't a sign of a healthy ecosystem, but of a pathological one.
No one can think that js has progressed substantially in the last three years, yet trying to build any project three years old without updates is so hard a rewrite is a reasonable solution.
> No one can think that js has progressed substantially in the last three years
Are we talking about the language, or the wider ecosystem?
If the latter, I think a lot of people would disagree. Bun is about three years old.
Other significant changes are Node.js being able to run TypeScript files without any optional flags, or being able to use require on ES Modules. I see positive changes in the ecosystem in recent years.
which returns 4155 results, though 727 of them are type packages.
Using these in commonjs code is trivial; they are automatically found by `require`. Unfortunately, system-installed packages are yet another casualty of the ESM transition ... there are ways to make it work but it's not automatic like it used to be.
The stable model usually implies that your app has to target both the old and the new distro version for a while. That is a bit too much to ask for some, unfortunately
There's a tradeoff and the assumption here (which I think is solid) is that there's more benefit from avoiding a supply chain attack by blindly (by default) using a dependency cooldown vs. avoiding a zero-day by blindly (by default) staying on the bleeding edge of new releases.
It's comparing the likelihood of an update introducing a new vulnerability to the likelihood of it fixing a vulnerability.
While the article frames this problem in terms of deliberate, intentional supply chain attacks, I'm sure the majority of bugs and vulnerabilities were never supply chain attacks: they were just ordinary bugs introduced unintentionally in the normal course of software development.
On the unintentional bug/vulnerability side, I think there's a similar argument to be made. Maybe even SemVer can help as a heuristic: a patch version increment is likely safer (less likely to introduce new bugs/regressions/vulnerabilities) than a minor version increment, so a patch version increment could have a shorter cooldown.
If I'm currently running version 2.3.4, and there's a new release 2.4.0, then (unless there's a feature or bugfix I need ASAP), I'm probably better off waiting N days, or until 2.4.1 comes out and fixes the new bugs introduced by 2.4.0!
Yep, that's definitely the assumption. However, I think it's also worth noting that zero-days, once disclosed, do typically receive advisories. Those advisories then (at least in Dependabot) bypass any cooldown controls, since the thinking is that a known vulnerability is more important to remediate than the open-ended risk of a compromised update.
> I'm sure the majority of bugs and vulnerabilities were never supply chain attacks: they were just ordinary bugs introduced unintentionally in the normal course of software development.
Yes, absolutely! The overwhelming majority of vulnerabilities stem from normal accidental bug introduction -- what makes these kinds of dependency compromises uniquely interesting is how immediately dangerous they are versus, say, a DoS somewhere in my network stack (where I'm not even sure it affects me).
You should also factor in that a zero-day often isn’t surfaced to be exploitable if you are using the onion model with other layers that need to be penetrated together. In contrast to a supply chain vulnerability that is designed to actively make outbound connections through any means possible.
Thank you. I was scanning this thread for anyone pointing this out.
The cooldown security scheme appears like some inverse "security by obscurity". Nobody could see a backdoor, therefor we can assume security. This scheme stands and falls with the assumed timelines. Once this assumption tumbles, picking a cooldown period becomes guess work. (Or another compliance box ticked.)
On the other side, the assumption can very well be sound, maybe ~90% of future backdoors can be mitigated by it. But who can tell. This looks like the survivorship bias, because we are making decisions based on the cases we found.
I’d estimate the vast majority of CVEs in third party source are not directly or indirectly exploitable. The CVSS scoring system assumes the worst case scenario the module is deployed in. We still have no good way to automate adjusting the score or even just figuring false positive.
The big problem is the Red Queen's Race nature of development in rapidly-evolving software ecosystems, where everyone has to keep pushing versions forward to deal with their dependencies' changes, as well as any actual software developments of their own. Combine that with the poor design decisions found in rapidly-evolving ecosystems, where everyone assumes anyting can be fixed in the next release, and you have a recipe for disaster.
I think there’s a much stronger argument for policies that both limit the number and complexity of dependencies. Don’t add it unless it’s highly focused (no “everything libraries” that pull in entire universes of their own) and carries a high level of value. A project’s entire dependency tree should be small and clean.
Libraries themselves should perhaps also take a page from the book of Linux distributions and offer LTS (long term support) releases that are feature frozen and include only security patches, which are much easier to reason about and periodically audit.
I've seen this argument made frequently. It's clearly a popular sentiment, but I can't help feel that it's one of those things that sounds nice in theory if you don't think about it too hard. (Also, cards on the table, I personally really like being able to pull in a tried-and-tested implementation of code to solve a common problem that's also used by in some cases literally millions of other projects. I dislike having to re-solve the same problem I have already solved elsewhere.)
Can you cite an example of a moderately-widely-used open source project or library that is pulling in code as a dependency that you feel it should have replicated itself?
What are some examples of "everything libraries" that you view as problematic?
Anything that pulled in chalk. You need a very good reason to emit escape sequences. The whole npm (and rust, python,..) ecosystem assumes that if it’s a tty, then it’s a full blown xterm-256color terminal. And then you need to pipe to cat or less to have sensible output.
So if you’re adding chalk, that generally means you don’t know jack about terminals.
The problem isn't the implementation of what I want to do. It's all of the implementations of things I never cared about doing. And the implementation of what I want to do that is soooo much more complex than it needs to be that I could easily have implemented it myself in less time.
The problem is also less about the implementation I want, it's about the 10,000 dependencies of things I don't really want. All of those are attack surface much larger than some simple function.
Most of your supply chain attack surface is social engineering attack surface. Doesn't really matter if I use Lodash, or 20 different single-function libraries, if I end up trusting the exact same people to not backdoor my server.
Of course, small libraries get a bad rap because they're often maintained by tons of different people, especially in less centralized ecosystems like npm. That's usually a fair assessment. But a single author will sometimes maintain 5, 10, or 20 different popular libraries, and adding another library of theirs won't really increase your social attack surface.
So you're right about "pull[ing] in universes [of package maintainers]". I just don't think complexity or number of packages are the metrics we should be optimizing. They are correlates, though.
(And more complex code can certainly contain more vulnerabilities, but that can be dealt with in the traditional ways. Complexity begets simplicity, yadda yadda; complexity that only begets complexity should obviously be eliminated)
I think AI nudges the economics more in this direction as well. Adding a non-core dependency has historically bought short-term velocity in exchange for different long-term maintenance costs. With AI, there are now many more cases where a first-party implementation becomes cheaper/easier/faster in both the short term and the long term.
Of course it's up to developers to weigh the tradeoffs and make reasonable choices, but now we have a lot more optionality. Reaching for a dependency no longer needs to be the default choice of a developer on a tight timeline/budget.
Let's have AI generate the same vulnerable code across hundreds of projects, most of which will remain vulnerable forever, instead of having those projects all depend on a central copy of that code that can be fixed and distributed once the issue gets discovered. Great plan!
A lot of projects are using dependencies, but are only using a small part of. Or are using them in a single place for a single usecase. Like bringing in formik (npm), but you only have one single form. Or moment, because you want to format a single date.
The lower level the dependency is, the more unjustifiable it is for it to have its own dependencies. This ought to be a point of competition between libraries and often is, at least in the c++ world
The think I find most odd about the constant pressure to update to the most recent and implied best version is that there is some implicit belief that software get's uniformly better with each release.
Bottom line those security bugs are not all from version 1.0 , and when you update you may well just be swapping known bugs for unknown bugs.
As has been said elsewhere - sure monitor published issues and patch if needed but don't just blindly update.
> Bottom line those security bugs are not all from version 1.0, and when you update you may well just be swapping known bugs for unknown bugs.
One great example of that is log4shell. If you were still using version 1.0 (log4j 1.x), you were not vulnerable, since the bug was introduced in version 2.0 (log4j 2.x). There were some known vulnerabilities in log4j 1.x, but the most common configuration (logging only to a local file or to the console, no remote logging or other exotic stuff) was not affected by any of them.
Funny you should mention that - amid the general log4j panic the mere presence of old version on disk would send IT security into a frenzy, irrespective of whether the old version was vulnerable or was deployed in a way where the vulnerability could be accessed.
I had to point out it was because we hadn't updated was the reason our stuff wasn't vulnerable.
The infamous log4j vulnerability was actually a feature when it was introduced. It was only several years later it was considered a security vulnerability. Countless of other techs has had the same problem, for example, ActiveX. Same with most downgrade attacks, they were most often considered a good thing (better compatibility) when introduced, only much later were such features considered non-good.
I remember this used to actually be the case, but that was many moons ago when you'd often wait a long time between releases. Or maybe the quality bar was lower generally, and it was just easier to raise it?
These days it seems most software just changes mostly around the margins, and doesn't necessarily get a whole lot better. Perhaps this is also a sign I'm using boring and very stable software which is mostly "done"?
The question is do you take control of your update cycle - or outsource it.
That's balancing the effort and risk of managing it yourself versus the risk and busy work generated from dependency churn.
What evergreen update policies have done is increase development velocity ( much easier to make breaking changes if you assume everyone is upto date everywhere ) - but also increases churn.
Feels like the tragedy of the commons: I don't want to look at the change, I don't want to take responsibility, somebody else will take care or it, I just have to wait.
Ok if this is an amazing advice and the entire ecosystem does that: just wait .... then what? We wait even more to be sure someone else is affected first?
Every time I see people saying you need to wait to upgrade it is like you are accumulating tech debt: the more you wait, the more painful the upgrade will be, just upgrade incrementally and be sure you have mitigations like 0 trust or monitoring to cut early any weird behavior.
You're not taking on any meaningful tech debt by waiting a week after a new version goes public to adopt it. As the OP says, there are services that scan popular open source tools for vulnerabilities as soon as they are released; even if a large percentage of the user base is waiting a week to update, many will still be caught in that period. And for various reasons some will still upgrade immediately.
You’re implicitly assuming that it’s exposure to downstream consumers that causes the malicious packages to be discovered, but we haven’t actually seen that in the last couple of major supply chain attacks. Instead it just buys time for the maintainers to undo the damage.
Even if less consumers will notice a compromise and report it, it still gives additional time for security researchers to analyze the packages, and for maintainers to notice themselves they got compromised
There are a lot of companies out there, that's scan packages and analyze them. Maintainers might notice a compromise, because a new release was published they didn't authorize. Or just during development, by getting all their bitcoin stolen ;)
The update tech debt tends not to compound or even accumulate at all. Usually you'd pay the same to update v1 to v2 as you would for v1 to v3. Maybe skipping v2 has a negative cost even.
I've seen this plenty of times: v1 of some library has one way of doing things, v2 of that library changes to a new incompatible way, and then v2.1 introduces a few extra changes to make it easier to port from the v1 way. If you wait a while, you have to do less work to update than if you had updated immediately.
One example is Python 3. After the first few Python 3.x releases, a few "useless" features were introduced to make it easier to port code from Python 2.7 (IIRC, things like reintroducing the u'...' syntax for unicode strings, which had been removed by Python 3.0 since normal '...' strings are now always unicode strings).
This is just completely wrong. If you are talking about a sizeable number of devices, you're not getting anything updated immediately even if you wanted to. You roll out to groups over a period of time because you don't want to break everything if there are unintended consequences. Your personal device? Sure whatever, but any fleet of devices absolutely does not get immediate updates across the board.
It's a good idea, but not without weak points, I think.
One of the classic scammer techniques is to introduce artificial urgency to prevent the victim from thinking clearly about a proposal.
I think this would be a weakness here as well: If enough projects adopt a "cooldown" policy, the focus of attackers would shift to manipulate projects into making an exception for "their" dependency and install it before the regular cooldown period elapsed.
How to do that? By playing the security angle once again: An attacker could make a lot of noise how a new critical vulnerability was discovered in their project and every dependant should upgrade to the emergency release as quickly as possible, or else - with the "emergency release" then being the actually compromised version.
I think a lot of projects would could come under pressure to upgrade, if the perceived vulnerability seems imminent and the only point for not upgrading is some generic cooldown policy.
Along those lines: If you're packaging an exploit, it's probably best to fix a bug while you're at it. That way people who want to remove their ugly workarounds will be motivated to violate the dependency cooldown.
Depends on the level of infiltration I guess. If the attacker managed to get themselves into a trusted position, as with the XZ backdoor, they could use the official communication channels of the project and possibility even file a CVE.
If it's "only" technical access, it would probably be harder.
One reason for cooldowns is not mentioned: maintainers often notice by themselves they got compromised.
The attacker will try to figure out when they are the least available: during national holidays, when they sleep, during conferences they attend, when they are on sick leave, personal time off, ...
Many projects have only a few or even only a single person, that's going to notice. They are often from the same country (time zone) or even work in the same company (they might all attend the same conference or company retreat weekend).
Readit News
This is indeed what's missing from the ecosystem at large. People seem to be under the impression that if a new release of software/library/OS/application is released, you need to move to it today. They don't seem to actually look through the changes, only doing that if anything breaks, and then proceed to upgrade because "why not" or "it'll only get harder in the future", neither which feel like solid choices considering the trade-offs.
While we've seen to already have known that it introduces massive churn and unneeded work, it seems like we're waking up to the realization that it is a security tradeoff as well, to stay at the edge of version numbers. Sadly, not enough tooling seems to take this into account (yet?).
What would happen from time to time was that an important reason did come up, but the team was now many releases behind. Whoever was unlucky enough to sign up for the project that needed the updated dependency now had to do all those updates of the dependency, including figuring out how they affected a bunch of software that they weren't otherwise going to work on. (e.g., for one code path, I need a bugfix that was shipped three years ago, but pulling that into my component affects many other code paths.) They now had to go figure out what would break, figure out how to test it, etc. Besides being awful for them, it creates bad incentives (don't sign up for those projects; put in hacks to avoid having to do the update), and it's also just plain bad for the business because it means almost any project, however simple it seems, might wind up running into this pit.
I now think of it this way: either you're on the dependency's release train or you jump off. If you're on the train, you may as well stay pretty up to date. It doesn't need to be every release the minute it comes out, but nor should it be "I'll skip months of work and several major releases until something important comes out". So if you decline to update to a particular release, you've got to ask: am I jumping off forever, or am I just deferring work? If you think you're just deferring the decision until you know if there's a release worth updating to, you're really rolling the dice.
(edit: The above experience was in Node.js. Every change in a dynamically typed language introduces a lot of risk. I'm now on a team that uses Rust, where knowing that the program compiles and passes all tests gives us a lot of confidence in the update. So although there's a lot of noise with regular dependency updates, it's not actually that much work.)
I think I wouldn't object to "Dependabot on a 2-week delay" as something that at least flags. However working in Go more than anything else it was often the case even so that dependency alerts were just an annoyance if they aren't tied to a security issue or something. Dynamic languages and static languages do not have the same risk profiles at all. The idea that some people have that all dependencies are super vital to update all the time and the casual expectation of a constant stream of vital security updates is not a general characteristic of programming, it is a specific characteristic not just of certain languages but arguably the community attached to those languages.
(What we really need is capabilities, even at a very gross level, so we can all notice that the supposed vector math library suddenly at version 1.43.2 wants to add network access, disk reading, command execution, and cryptography to the set of things it wants to do, which would raise all sorts of eyebrows immediately, even perhaps in an automated fashion. But that's a separate discussion.)
(Literally at one place we built a SPA frontend that was embedded in the device firmware as a static bundle, served to the client and would then talk to a small API server. And because these NodeJS types liked to have libraries reused for server and frontend, we would get endless "vulnerability reports" - but all of this stuff only ever ran in the clients browser!)
that's generally true, no?
of course waiting a few days/weeks should be the minimum unless there's a CVE (or equivalent) that's applies
In the end, I decided to implement a lightweight PHP 5 relay to translate SQL requests so the MSSQL server could still be accessed. The university stuffs were quite satisfied with my work. But I really felt guilty for the next guy who will touch this setup. So I still didn’t quite make it back onto the release train… as that PHP 5 relay counts.
Most tooling (e.g. Dependabot) allows you to set an interval between version checks. What more could be done on that front exactly? Devs can already choose to check less frequently.
Deleted Comment
The practical problem with this is that many large organizations have a security/infosec team that mandates a "zero CVE" posture for all software.
Where I work, if our infosec team's scanner detect a critical vulnerability in any software we use, we have 7 days to update it. If we miss that window we're "out of compliance" which triggers a whole process that no one wants to deal with.
The path of least resistance is to update everything as soon as updates are available. Consequences be damned.
Come 2027-12, the Cyber Resilience Act enters full enforcement. The CRA mandates a "duty of care" for the product's lifecycle, meaning if a company blindly updates a dependency to clear a dashboard and ships a supply-chain compromise, they are on the hook for fines up to €15M or 2.5% of global turnover.
At that point, yes, there is a sense in which the blind update strategy you described becomes a legal liability. But don't miss the forest for the trees, here. Most software companies are doing zero vetting whatsoever. They're staring at the comet tail of an oncoming mass extinction event. The fact that you are already thinking in terms of "assess impact" vs. "blindly patch" already puts your workplace significantly ahead of the market.
We had like 1 or 2 crash-patches in the past - Log4Shell was one of them, and blocking an API no matter what in a component was another one.
In a lot of other cases, you could easily wait a week or two for directly customer facing things.
The solution is to fire those teams.
Browsers get a lot of unknown input, so they have to update often.
A Weather app is likely to only get input from one specific site (controlled by the app developers), so it should be relatively safe.
Yes
"Only then do you need to update that specific dependency right away."
Big no. If you do that it is guaranteed one day you miss a vulnerability that hurts you.
To frame it differently: What you propose sounds good in theory but in practice the effort to evaluate vulnerabilities against your product will be higher than the effort to update plus taking appropriate measures against supply chain attacks.
Most of them assume what if they are working on some public accessible website then 99% of the people and orgs in the world are running nothing but some public accessible website.
Dependencies are good. Churn is bad.
Nobody is proposing a system that utterly and completely locks you out of all updates if they haven't aged enough. There is always going to be an override switch.
It's a shame some ecosystems move waaay too fast, or don't have a good story for having distro-specific packages. For example, I don't think there are Node.js libraries packaged for Debian that allow you to install them from apt and use it in projects. I might be wrong.
An eco system moving too quickly, when it isn't being fundamentally changed, isn't a sign of a healthy ecosystem, but of a pathological one.
No one can think that js has progressed substantially in the last three years, yet trying to build any project three years old without updates is so hard a rewrite is a reasonable solution.
Are we talking about the language, or the wider ecosystem?
If the latter, I think a lot of people would disagree. Bun is about three years old.
Other significant changes are Node.js being able to run TypeScript files without any optional flags, or being able to use require on ES Modules. I see positive changes in the ecosystem in recent years.
Web search shows some: https://packages.debian.org/search?keywords=node&searchon=na... (but also shows "for optimizing reasons some results might have been suppressed" so might not be all)
Although probably different from other distros, Arch for example seems to have none.
Using these in commonjs code is trivial; they are automatically found by `require`. Unfortunately, system-installed packages are yet another casualty of the ESM transition ... there are ways to make it work but it's not automatic like it used to be.
It's comparing the likelihood of an update introducing a new vulnerability to the likelihood of it fixing a vulnerability.
While the article frames this problem in terms of deliberate, intentional supply chain attacks, I'm sure the majority of bugs and vulnerabilities were never supply chain attacks: they were just ordinary bugs introduced unintentionally in the normal course of software development.
On the unintentional bug/vulnerability side, I think there's a similar argument to be made. Maybe even SemVer can help as a heuristic: a patch version increment is likely safer (less likely to introduce new bugs/regressions/vulnerabilities) than a minor version increment, so a patch version increment could have a shorter cooldown.
If I'm currently running version 2.3.4, and there's a new release 2.4.0, then (unless there's a feature or bugfix I need ASAP), I'm probably better off waiting N days, or until 2.4.1 comes out and fixes the new bugs introduced by 2.4.0!
> I'm sure the majority of bugs and vulnerabilities were never supply chain attacks: they were just ordinary bugs introduced unintentionally in the normal course of software development.
Yes, absolutely! The overwhelming majority of vulnerabilities stem from normal accidental bug introduction -- what makes these kinds of dependency compromises uniquely interesting is how immediately dangerous they are versus, say, a DoS somewhere in my network stack (where I'm not even sure it affects me).
The cooldown security scheme appears like some inverse "security by obscurity". Nobody could see a backdoor, therefor we can assume security. This scheme stands and falls with the assumed timelines. Once this assumption tumbles, picking a cooldown period becomes guess work. (Or another compliance box ticked.)
On the other side, the assumption can very well be sound, maybe ~90% of future backdoors can be mitigated by it. But who can tell. This looks like the survivorship bias, because we are making decisions based on the cases we found.
Libraries themselves should perhaps also take a page from the book of Linux distributions and offer LTS (long term support) releases that are feature frozen and include only security patches, which are much easier to reason about and periodically audit.
Can you cite an example of a moderately-widely-used open source project or library that is pulling in code as a dependency that you feel it should have replicated itself?
What are some examples of "everything libraries" that you view as problematic?
So if you’re adding chalk, that generally means you don’t know jack about terminals.
The problem is also less about the implementation I want, it's about the 10,000 dependencies of things I don't really want. All of those are attack surface much larger than some simple function.
Of course, small libraries get a bad rap because they're often maintained by tons of different people, especially in less centralized ecosystems like npm. That's usually a fair assessment. But a single author will sometimes maintain 5, 10, or 20 different popular libraries, and adding another library of theirs won't really increase your social attack surface.
So you're right about "pull[ing] in universes [of package maintainers]". I just don't think complexity or number of packages are the metrics we should be optimizing. They are correlates, though.
(And more complex code can certainly contain more vulnerabilities, but that can be dealt with in the traditional ways. Complexity begets simplicity, yadda yadda; complexity that only begets complexity should obviously be eliminated)
Of course it's up to developers to weigh the tradeoffs and make reasonable choices, but now we have a lot more optionality. Reaching for a dependency no longer needs to be the default choice of a developer on a tight timeline/budget.
Limiting the number of dependencies, but then rewriting them in your own code, will also increase the maintenance burden and compile times
Deleted Comment
Bottom line those security bugs are not all from version 1.0 , and when you update you may well just be swapping known bugs for unknown bugs.
As has been said elsewhere - sure monitor published issues and patch if needed but don't just blindly update.
One great example of that is log4shell. If you were still using version 1.0 (log4j 1.x), you were not vulnerable, since the bug was introduced in version 2.0 (log4j 2.x). There were some known vulnerabilities in log4j 1.x, but the most common configuration (logging only to a local file or to the console, no remote logging or other exotic stuff) was not affected by any of them.
I had to point out it was because we hadn't updated was the reason our stuff wasn't vulnerable.
These days it seems most software just changes mostly around the margins, and doesn't necessarily get a whole lot better. Perhaps this is also a sign I'm using boring and very stable software which is mostly "done"?
That's balancing the effort and risk of managing it yourself versus the risk and busy work generated from dependency churn.
What evergreen update policies have done is increase development velocity ( much easier to make breaking changes if you assume everyone is upto date everywhere ) - but also increases churn.
Ok if this is an amazing advice and the entire ecosystem does that: just wait .... then what? We wait even more to be sure someone else is affected first?
Every time I see people saying you need to wait to upgrade it is like you are accumulating tech debt: the more you wait, the more painful the upgrade will be, just upgrade incrementally and be sure you have mitigations like 0 trust or monitoring to cut early any weird behavior.
There are a lot of companies out there, that's scan packages and analyze them. Maintainers might notice a compromise, because a new release was published they didn't authorize. Or just during development, by getting all their bitcoin stolen ;)
I've seen this plenty of times: v1 of some library has one way of doing things, v2 of that library changes to a new incompatible way, and then v2.1 introduces a few extra changes to make it easier to port from the v1 way. If you wait a while, you have to do less work to update than if you had updated immediately.
One example is Python 3. After the first few Python 3.x releases, a few "useless" features were introduced to make it easier to port code from Python 2.7 (IIRC, things like reintroducing the u'...' syntax for unicode strings, which had been removed by Python 3.0 since normal '...' strings are now always unicode strings).
Not enough to accumulate tech debt, enough to mitigate the potential impact of any supply-chain vulnerability.
One of the classic scammer techniques is to introduce artificial urgency to prevent the victim from thinking clearly about a proposal.
I think this would be a weakness here as well: If enough projects adopt a "cooldown" policy, the focus of attackers would shift to manipulate projects into making an exception for "their" dependency and install it before the regular cooldown period elapsed.
How to do that? By playing the security angle once again: An attacker could make a lot of noise how a new critical vulnerability was discovered in their project and every dependant should upgrade to the emergency release as quickly as possible, or else - with the "emergency release" then being the actually compromised version.
I think a lot of projects would could come under pressure to upgrade, if the perceived vulnerability seems imminent and the only point for not upgrading is some generic cooldown policy.
If it's "only" technical access, it would probably be harder.
The attacker will try to figure out when they are the least available: during national holidays, when they sleep, during conferences they attend, when they are on sick leave, personal time off, ...
Many projects have only a few or even only a single person, that's going to notice. They are often from the same country (time zone) or even work in the same company (they might all attend the same conference or company retreat weekend).