I feel like there's a lot of misunderstanding of this issue in the software community, because primarily, supply chain risk isn't a software or engineering issue. It's a governance issue.
Someone doesn't have to be a bad actor for a project to have supply chain risk. Nor do all who evaluate supply chain risk have the same security posture and evaluate risks the same as others might. The DoD likely has a very different set of risks they evaluate against for their security posture than you do.
Most supply chain risks are not an indictment of somebody's code or somebody's character. A lot of one person projects are risky just because they're only one person. Having a bus factor of one is a supply chain risk in and of itself.
And while most people don't prepare for war while choosing their packages, it's not unreasonable for a military to do so. During a war, the ability for people to govern themselves and their own projects often changes dramatically, even in democratic countries. It is entirely routine for countries to require cooperation by the force of law in war time, even the US can and has forced private companies to cooperate with war efforts. This is probably not in the security posture calculation for most of us. But it is for some.
How about: commit your dependency lockfiles, make sure they use content-addressing cryptographic checksums like Cargo.lock does.
This is also needed for both reproducible builds and SBOMs.
If you commit the actual source code you're making things worse, because it makes coordinated source code review efforts a lot harder. Also patch management with actual vendored source code is terrible.
> And while most people don't prepare for war while choosing their packages, it's not unreasonable for a military to do so. During a war, the ability for people to govern themselves and their own projects often changes dramatically, even in democratic countries. It is entirely routine for countries to require cooperation by the force of law in war time, even the US can and has forced private companies to cooperate with war efforts. This is probably not in the security posture calculation for most of us. But it is for some.
Reading this I'm really hoping the DOD maintains mirrors of GitHub projects that are vital to them.
Still, we had a lot of security issues of people injecting themselves into these one-man projects.
It is ridiculous to force these people to do anything because you were to lazy to build the foundations of your infrastructure yourself, especially if you care about self-reliance.
Huh? The DoD would not have used the package if they hadn't read every line, locked it down for updates, and were ready to patch it themselves if needed. Can you really imagine in a war they'd be like "damn, if only there were a second person we also don't trust at all to do this work for us cause otherwise we'd just be SOL"
It mostly doesn't work like that even for closed source in DoD. They have to weigh the risk against the very high cost of mitigating the risk. Their resources are large but not infinite.
Even if they trust the developer they may not trust their process. There are many cases of trusted developers having their development environments compromised such that bad actors were able to insert modifications into source trees, in commits signed by the developer. Most code is not developed in anything remotely resembling a high security context.
I don't know where you're working, maybe you work in some secret lab where everything is air-gapped and not even the pigeons are allowed within a mile of the facility. In which case, what the hell are you doing commenting on a public message board?
That is absolutely not how DoD works. The vast majority of code is contracted out. Nobody from DoD side is reading any of the code. It's all a series of affidavits and audits for configuration management process. Vendors assert everything's cool. Failed audits lead to fines or revocation of access. And the audits check up on documentation and config. They don't dig into code.
At no point in time is anyone, anywhere, in this process reading every single line of code. Not even A single line of code. I doubt they even read the Software Bill of Materials we're supposed to generate, because I've never heard any feedback on any of it.
I think you're seriously overestimating the amount of work the DoD will use... It really depends on what you are working on and where it will be used. I've worked on govt adjacent, military and banking projects... The most locked down in terms of packages I can use have been banks. In one case, a lawyer had to review (mostly licensing) every package that got added in to the local npm mirror for allowed internal use.. and another review for every version bump. Then of course, the (one) guy retires and there's no reviews for a month (so much for the launch date).
I've also been in a sealed environment, where I literally had to hand copy jQuery from an internet connected computer on one side of the room to an internal dev computer on the other side of the room... no disks, usb drives, etc allowed. That was a few days of "fun."
> So while NPM has over 4 million single person projects, they have about 900,000 maintainers for those 4 million single person projects. This will be an important data point at the end.
Am I missing something or was it not, in fact, an important data point at the end?
I didn't see it explicitly stated, but I think it supports the "overworked" part of this statement:
> Open source, the thing that drives the world, the thing Harvard says has an economic value of 8.8 trillion dollars (also a big number). Most of it is one person. And I can promise you not one of those single person projects have the proper amount of resources they need. If you want to talk about possible risks to your supply chain, a single maintainer that’s grossly underpaid and overworked. That’s the risk. The country they are from is irrelevant.
Has anyone seen any stats on what happens to a single maintainer project when said person is hit by a bus (or meets some other demise)? With that many data points, there should be enough of them by now to study it.
Is the project taken over by another, single developer? Is it replaced by a similar project? Does it just go away?
It depends. More common than getting hit by a bus is that the maintainer loses interest, or doesn't have the time to put into it anymore. When that happens I've seen all of the following happen:
* Someone forks the project, and eventually the fork replaces the original
* Another, possibly new, project that fills the same niche becomes more popular, and eventually replaces most usages of the first project.
* The original maintainer hands off maintenance to someone else.
* People keep using it, even though it is no longer maintained, and maybe make their own forks to fix issues they have, but none of the forks really catch on
One of the strengths of OSS is that if the developer disappears, or goes rogue, or changes the license terms, someone can fork the project and keep it going. With proprietary software, if the company (or individual) who makes it disappears, or decides to discontinue it, or change the terms to something unacceptable, you are just out of luck. Hopefully, you can find a competing product that meets your needs.
Definitely seen this a lot in the JS/NPM ecosystem... You go searching for a module that does $thing... you find about 10, you sort and look at say the 3 most recently published an the 3-5 most downloaded/popular... is the repo open (github, usually), are there a lot of old issues left lingering with an old last publish date? Might take a passive look at the codebase to see if I can grok it and fix any issues I find if needed.
Choose what I feel is the best option. Trying to avoid dead packages, but not afraid to deal with older packages if they aren't just stale, but functionally complete. The shift towards ES import statements and TypeScript defs has also influenced my selection process.
I've seen plenty of cases where either a fork or new option effectively takes over. A lot of people are leaning towards Zod over Yue or Hono over Express. There's instances where the dev goes off the rails like with Faker and the community comes together to fork a solution.
All of the above examples definitely happen in practice. I'm guessing many packages all over the place have replaced various dependencies over the years.
I would love to see a diligently researched episodic series, every episode covering the transition of a popular open-source library/tool/app/site from one maintainer to the next.
I bought ASIO Link Pro (software) something like 10 years ago to help route virtual audio devices on my system. The author sadly died and eventually the license key server went offline rendering it unable to start. His nephew looked into it and eventually made the tool free after a year or 2.
I stopped using it after the license server went offline because I still had to record videos. I ended up solving my problem with hardware, but that tool was extremely helpful when I used it for years. It was around $40 at the time. It's one of the few pieces of software I've purchased and felt really happy about it.
I suspect this is the case for the majority of open source software. I have a handful of tiny projects. I don't think anyone will keep them alive after I die. But I guess we should make a distinction based on popularity or something. My top four projects have only 675, 363, 122, and 96 stars.
Closest example I could think of would be Hans Reiser/Reiserfs. It's a more sordid story than just getting hit by a bus, though. Ultimately the project just died.
I don't think this is a good example though as the "sordid" part also made the project toxic for anything that might have otherwise chosen to take it on.
I don't know about any broader statistics, but in my personal experience, I see all three of those. I think it's mostly a function of how large the user base is, how complicated the code base is, and whether or not there are any substitutes.
I think this is one thing that people fail to consider: if the code is open source, though it may take time to understand, worst case scenario you can just fork it.
- Hans Reiser, maintainer of ReiserFS. I think very few people use ReiserFS these days.
- Ian Murdock, creator of the Debian distribution. Debian lives on, but the project was also set up specifically to distribute maintenance.
- Jim Weirich, creator of the Rake build tool. I'm not a Rubyist so I don't know how it was affected, but I assume it's such a big part of Ruby other people took over.
- Peter Hintjens, co-creator of ZeroMQ. From what I understand, Hintjens was never the main developer but an active promoter. The project lives on as far as I know.
- Terry Davis, creator of TempleOS. I think development on TempleOS stopped.
IMO, it has a lot to do with usage and the availability of alternatives. With ReiserFS, there were a lot of alternatives, both available at the time or announced shortly. While ReiserFS pioneered a lot of ideas, many of them showed up in alternatives fairly quickly. TempleOS is had a pretty limited user base.
I’ve seen many projects in the Clojure ecosystem get picked up and maintained by other folks. The key was always that the projects had an established user base of some notable size and something distinctive about them that made switching to other alternatives less desirable than continuing to push forward with a new and possibly more mundane maintainer and feature schedule. I’ve also seen a lot of “abandonware.”
Reiserfs died because alternatives, like ext3/ext4 and btrfs, became readily available.
TempleOS has a fork called ZealOS. Terry Davis really was the "Wesley Willis of programming", and he had friends and fans worldwide, some of whom have taken up TempleOS development under the ZealOS banner.
Definitely varies with language/runtime/library choice. I have no problem using a clojure library that hasn’t been touched in 5 years. But back when I had a gatsby site (static site generator for react) I would end up in the dependency hell after literally a month of not touching it
But if you had a "perfect" piece of software that used Log4j in 2020, it wouldn't have been perfect for long.
Unfortunately, there's a lot of reasons that software needs maintenance, even if it was thought to be perfect when it was originally written.
Hardware changes. The software landscape changes. Dependencies are deprecated, or are found to have their own problems. Vulnerabilities are discovered. Vulnerabilities are found that aren't even the fault of your software, maybe they are a flaw in the hardware your software runs on, and the only way to fix it is via a software mitigation. These are all real things that happen to otherwise perfect software.
It is true of Solitaire, Minesweeper, Calculator, and Notepad, and probably about the same number of programs on other OSes. (Notepad has recently had an important expansion of functionality, but it didn't NEED that change.)
It's also true of some dinosaurs I have on my system, that copy DVDs and so forth.
It's not true of most other applications, nor can it be true, unless the app works in a sealed, unchanging environment.
Even then... Voyager 2 recently required a software upgrade, IIRC.
You'd think so, but you make something then it doesn't work on a new version of windows, or it doesn't work on a new version of python because one of your dependencies isn't available for that version of python, or it doesn't work on linux if it doesn't have a specific version of packages, or it doesn't work on the browser because they're ditching manifest v2, or it doesn't work on android because you need to provide more personal information or your app will be unpublished.
At this point I have a feeling "perfect" software only exists in hardware like consoles where updates just stop one day.
LOL. As soon as Python 3.8 is deprecated/replaced by Python 3.9+ in most systems, python packages that depend on old APIs become useless until updated. Any half decent software engineer understands this.
The DOD is one of the world's largest organizations. There are people there who do things like publish newsletters and put up webpages for people like boy scouts to arrange tour bases. It is totally fine to use Node for things like that.
Those systems are not connected to the systems that fire missiles. If the sign up page for the 4th of July fireworks announcement gets vandalized, it isn't really an issue.
The DoD is very efficient at finding something they are getting for free and convincing everyone it's in their best interest to pay a team of contractors for it.
I've heard good things about work done by this guy Linus. I'm pretty sure that I've used his work.
I think he comes from a country that borders Russia, so should we be worried?
I've done OSS for decades; mostly by myself, but sometimes, in teams of volunteers.
If anyone has any experience, working in teams of volunteers, it can be ... challenging.
It can definitely work, but not as often as you'd think. If it works, there's usually some "BDFL," or a common goal that has everyone on the same beam. In my case, it was usually the latter.
Not only that, but Linus's parents were politically active communists and young Linus was a pioneer (like a boy scout but for communists). His father also lived in Moscow for several years on two separate occasions.
Being a Young Pioneer or joining the Komsomol was not officially mandatory, but it functioned as a gatekeeper for any kind of advancement. Party membership operated the same way.
So, by themselves, they don't tell you whether the person in question is a communist.
I don't think Russia (or China, either) has been truly communist, in a long time.
Not sure there are any real communist nations left. It's one of those ideologies that looks good on paper, but falls apart, as soon as humans get added to the soup.
Idealists never seem to account for base human nature.
Not anymore, but that was not always the case. He just has an extremely strong will, and a force of personality, that was able to shepherd the project through its nascent challenges.
I have founded fairly important projects (nowhere near on the scale of his work, though), but I don't have the force of personality he does, so tossing the keys to a new team, and walking away, is what worked for me.
Readit News
Someone doesn't have to be a bad actor for a project to have supply chain risk. Nor do all who evaluate supply chain risk have the same security posture and evaluate risks the same as others might. The DoD likely has a very different set of risks they evaluate against for their security posture than you do.
Most supply chain risks are not an indictment of somebody's code or somebody's character. A lot of one person projects are risky just because they're only one person. Having a bus factor of one is a supply chain risk in and of itself.
And while most people don't prepare for war while choosing their packages, it's not unreasonable for a military to do so. During a war, the ability for people to govern themselves and their own projects often changes dramatically, even in democratic countries. It is entirely routine for countries to require cooperation by the force of law in war time, even the US can and has forced private companies to cooperate with war efforts. This is probably not in the security posture calculation for most of us. But it is for some.
This is also needed for both reproducible builds and SBOMs.
If you commit the actual source code you're making things worse, because it makes coordinated source code review efforts a lot harder. Also patch management with actual vendored source code is terrible.
Reading this I'm really hoping the DOD maintains mirrors of GitHub projects that are vital to them.
It is ridiculous to force these people to do anything because you were to lazy to build the foundations of your infrastructure yourself, especially if you care about self-reliance.
Even if they trust the developer they may not trust their process. There are many cases of trusted developers having their development environments compromised such that bad actors were able to insert modifications into source trees, in commits signed by the developer. Most code is not developed in anything remotely resembling a high security context.
That is absolutely not how DoD works. The vast majority of code is contracted out. Nobody from DoD side is reading any of the code. It's all a series of affidavits and audits for configuration management process. Vendors assert everything's cool. Failed audits lead to fines or revocation of access. And the audits check up on documentation and config. They don't dig into code.
At no point in time is anyone, anywhere, in this process reading every single line of code. Not even A single line of code. I doubt they even read the Software Bill of Materials we're supposed to generate, because I've never heard any feedback on any of it.
I've also been in a sealed environment, where I literally had to hand copy jQuery from an internet connected computer on one side of the room to an internal dev computer on the other side of the room... no disks, usb drives, etc allowed. That was a few days of "fun."
That's a lot of faith in military intelligence.
Am I missing something or was it not, in fact, an important data point at the end?
> Open source, the thing that drives the world, the thing Harvard says has an economic value of 8.8 trillion dollars (also a big number). Most of it is one person. And I can promise you not one of those single person projects have the proper amount of resources they need. If you want to talk about possible risks to your supply chain, a single maintainer that’s grossly underpaid and overworked. That’s the risk. The country they are from is irrelevant.
Is the project taken over by another, single developer? Is it replaced by a similar project? Does it just go away?
* Someone forks the project, and eventually the fork replaces the original
* Another, possibly new, project that fills the same niche becomes more popular, and eventually replaces most usages of the first project.
* The original maintainer hands off maintenance to someone else.
* People keep using it, even though it is no longer maintained, and maybe make their own forks to fix issues they have, but none of the forks really catch on
One of the strengths of OSS is that if the developer disappears, or goes rogue, or changes the license terms, someone can fork the project and keep it going. With proprietary software, if the company (or individual) who makes it disappears, or decides to discontinue it, or change the terms to something unacceptable, you are just out of luck. Hopefully, you can find a competing product that meets your needs.
Choose what I feel is the best option. Trying to avoid dead packages, but not afraid to deal with older packages if they aren't just stale, but functionally complete. The shift towards ES import statements and TypeScript defs has also influenced my selection process.
I've seen plenty of cases where either a fork or new option effectively takes over. A lot of people are leaning towards Zod over Yue or Hono over Express. There's instances where the dev goes off the rails like with Faker and the community comes together to fork a solution.
All of the above examples definitely happen in practice. I'm guessing many packages all over the place have replaced various dependencies over the years.
And that's why I don't run Netflix.
I bought ASIO Link Pro (software) something like 10 years ago to help route virtual audio devices on my system. The author sadly died and eventually the license key server went offline rendering it unable to start. His nephew looked into it and eventually made the tool free after a year or 2.
I stopped using it after the license server went offline because I still had to record videos. I ended up solving my problem with hardware, but that tool was extremely helpful when I used it for years. It was around $40 at the time. It's one of the few pieces of software I've purchased and felt really happy about it.
https://github.com/DirkoAudio/ASIOLinkProFIX
I've been using it for over a year on Windows 10 and it works great.
- Hans Reiser, maintainer of ReiserFS. I think very few people use ReiserFS these days.
- Ian Murdock, creator of the Debian distribution. Debian lives on, but the project was also set up specifically to distribute maintenance.
- Jim Weirich, creator of the Rake build tool. I'm not a Rubyist so I don't know how it was affected, but I assume it's such a big part of Ruby other people took over.
- Peter Hintjens, co-creator of ZeroMQ. From what I understand, Hintjens was never the main developer but an active promoter. The project lives on as far as I know.
- Terry Davis, creator of TempleOS. I think development on TempleOS stopped.
I’ve seen many projects in the Clojure ecosystem get picked up and maintained by other folks. The key was always that the projects had an established user base of some notable size and something distinctive about them that made switching to other alternatives less desirable than continuing to push forward with a new and possibly more mundane maintainer and feature schedule. I’ve also seen a lot of “abandonware.”
So, it’s a bit of a mixed bag.
TempleOS has a fork called ZealOS. Terry Davis really was the "Wesley Willis of programming", and he had friends and fans worldwide, some of whom have taken up TempleOS development under the ZealOS banner.
If there is a major change (e.g., Python 3, React Native new arch), they are replaced/forked.
updating is a systemic issue, not a per-project matter
But if you had a "perfect" piece of software that used Log4j in 2020, it wouldn't have been perfect for long.
Unfortunately, there's a lot of reasons that software needs maintenance, even if it was thought to be perfect when it was originally written.
Hardware changes. The software landscape changes. Dependencies are deprecated, or are found to have their own problems. Vulnerabilities are discovered. Vulnerabilities are found that aren't even the fault of your software, maybe they are a flaw in the hardware your software runs on, and the only way to fix it is via a software mitigation. These are all real things that happen to otherwise perfect software.
It is true of Solitaire, Minesweeper, Calculator, and Notepad, and probably about the same number of programs on other OSes. (Notepad has recently had an important expansion of functionality, but it didn't NEED that change.)
It's also true of some dinosaurs I have on my system, that copy DVDs and so forth.
It's not true of most other applications, nor can it be true, unless the app works in a sealed, unchanging environment.
Even then... Voyager 2 recently required a software upgrade, IIRC.
At this point I have a feeling "perfect" software only exists in hardware like consoles where updates just stop one day.
Hell, even sysvinit had some big updates recently.
A perfected software that had existed >5 years with zero updates, tweaks, ports, or fixes?
[0] https://github.com/11ty/eleventy/graphs/contributors
I might be wrong but npm etc feels like a very large attack surface.
The DOD is one of the world's largest organizations. There are people there who do things like publish newsletters and put up webpages for people like boy scouts to arrange tour bases. It is totally fine to use Node for things like that.
Those systems are not connected to the systems that fire missiles. If the sign up page for the 4th of July fireworks announcement gets vandalized, it isn't really an issue.
That's an understatement if there ever was one.
https://en.wikipedia.org/wiki/List_of_largest_employers
Deleted Comment
I think he comes from a country that borders Russia, so should we be worried?
I've done OSS for decades; mostly by myself, but sometimes, in teams of volunteers.
If anyone has any experience, working in teams of volunteers, it can be ... challenging.
It can definitely work, but not as often as you'd think. If it works, there's usually some "BDFL," or a common goal that has everyone on the same beam. In my case, it was usually the latter.
Not only that, but Linus's parents were politically active communists and young Linus was a pioneer (like a boy scout but for communists). His father also lived in Moscow for several years on two separate occasions.
Being a Young Pioneer or joining the Komsomol was not officially mandatory, but it functioned as a gatekeeper for any kind of advancement. Party membership operated the same way.
So, by themselves, they don't tell you whether the person in question is a communist.
Not sure there are any real communist nations left. It's one of those ideologies that looks good on paper, but falls apart, as soon as humans get added to the soup.
Idealists never seem to account for base human nature.
I have founded fairly important projects (nowhere near on the scale of his work, though), but I don't have the force of personality he does, so tossing the keys to a new team, and walking away, is what worked for me.
Deleted Comment