Excellent summary of the events, with all the links in one place. This is the perfect resource for anyone who want to catch up, and also to learn about how such things (especially social engineering) unfold in the wild, out in the open.
One thing that could be added, for the sake of completeness: in the part "Attack begins", toward the end, when they are pushing for updating xz in the major distros, Ubuntu and Debian are mentioned but not Fedora.
Looks like the social engineering/pressuring for Fedora started at least weeks before 2024 March 04, according to a comment by @rwmj on HN [1]. I also found this thread on Fedora's devel list [2], but didn't dig too much.
It would be intetesting if Lasse Collin published his off-list interactions with 'Jia Tan' and any of the other pseudonyms, to get an even better angle on the social engineering parts. Apparently a large part of the campaign was via private channels to Lasse.
Collin has been writing more extensively on IRC. A screenshot of one of his posts can be seen in [this YouTube video](https://youtu.be/0pT-dWpmwhA?t=1158).
He notes that while he was unsatisfied with some of the changes Tan introduced, Tan was nonetheless extremely helpful.
Missing the whole Fedora timeline. I was emailed by "Jia Tan" between Feb 27 and Mar 27, in a partially successful attempt to get the new xz into Fedora 40 & 41. Edit: I emailed Russ with the details.
I wondered about this. I saw the note at the bottom "RedHat announces that the backdoored xz shipped in Fedora Rawhide and Fedora Linux 40 beta" but saw nothing in the timeline explaining when/how it made it into Fedora.
That's a great first step, but ready your pitchforks for this next take, because the next step is to completely eliminate Turing-complete languages and arbitrary I/O access from standard build systems. 99.9% of all projects have the capability to be built with trivial declarative rulesets.
Java's Maven is an interesting case study, as it tried to be this:. A standard project layout, a standard dependency mechanism, pom.xml as standard metadata file, and a standard workflow with standard target(clean/compile/test/deploy). Plugins for what's left.
There might have been a time where it worked, but people started to use plugins for all kinds of reasons, quite good ones in most cases. Findbugs, code coverage, source code generation,...
Today, a maven project without plugins is rare. Maven brought us 95%, but there is a long tail left to cover.
In this case, I think the GP is absolutely right. If you look at the infamous patch with a "hidden" dot, you may think "any C linter should catch that syntax error and immediately draw suspicion." But the thing is, no linter at the moment exists for analyzing strings in a CMakeLists.txt file or M4 macro. Moreover, this isn't something one can reliably do runtime detection for, because there are plenty of legitimate reasons that program could fail to compile, but our tooling does not have a way to clearly communicate the syntax error being the reason for a compilation failure.
I think that what we need is good sandboxing. All a sandboxed Turing-complete language can do is perform I/O on some restricted area, burn CPU and attempt to escape the sandbox.
I would like to see this on the language level, not just on the OS level.
This doesn't read as a technical failure to me. This was 99% social engineering. I understand that the build system was used a vector, but eliminating that vector doesn't mean all doors are closed. The attacker took advantage of someone having trouble.
While I do agree that M4 is not great I don't believe any alternative would have prevented this attack: you could try translating that build-to-host file to say python with all the evals and shell oneliners and it would still not be immediately obvious for a distro package maintainer doing a casual review after work: for them it would look just like your average weekend project hacky code. Even if you also translated the oneliners it wouldn't be immediately obvious if you were not suspicious. My point is, you could write similarly obfuscated code in any language.
What's inscrutable code? Was it m4 or sh or the combination of the two?
Who will pay for all the rewriting you want done? Or even just for the new frameworks that are "scrutable"? How do we guarantee that the result is not inscrutable to you or others?
There is so much knee-jerking in this xz debacle.
(And I say this / ask these questions with no love for autoconf/m4/sh.)
I do think we have enough eyeballs at this point that we should stop entertaining the Dancing Bear in low level libraries and start insisting on crisp, self-explaining code. There are a lot of optimizations pushed into compilers these days, and there are a lot of architectural changes that can make things fast without making them inscrutable.
We should be moving from No Obvious Bugs to Obviously No Bugs (Tony Hoare).
That would be an improvement for sure... but this is not fundamentally a technical problem.
Reading the timeline, the root cause is pure social engineering. The technical details could be swapped out with anything. Sure there are aspects unique to xz that were exploited such as the test directory full of binaries, but that's just because they happened to be the easiest target to implement their changes in an obfuscated way, that misses the point - once an attacker has gained maintainer-ship you are basically screwed - because they will find a way to insert something, somehow, eventually, no matter how many easy targets for obfuscation are removed.
Is the real problem here in handing over substantial trust to an anonymous contributor? If a person has more to lose than maintainership then would this have happened?
That someone can weave their way into a project under a pseudonym and eventually gain maintainership without ever risking their real reputation seems to set up quite a risk free target for attackers.
> Is the real problem here handing over substantial trust to an anonymous contributor?
Unless there's some practical way of comprehensively solving The Real Problem Here, it makes a lot of sense to consider all reasonable mitigations, technical, social or otherwise.
> If a person has more to lose than maintainership then would this have happened?
I guess that's one possible mitigation, but what exactly would that look like in practice? Good luck getting open source contributors to accept any kind of liability. Except for the JiaTans of the world, who will be the first to accept it since they will have already planned to be untouchable.
Agreed it's mainly a social engineering problem, BUT also can be viewed as a technical problem, if a sufficiently advanced fuzzer could catch issues like this.
It could also be called an industry problem, where we rely on other's code without running proper checks. This seems to be an emerging realization, with services like socket.dev starting to emerge.
Autotools-based build infra is always crufty, fragile, slow, and fugly. In a lot of cases, it sucks because it's what we have right now without doing the work to replace it with something equally flexible.
Build infrastructure should be minimal, standardized, and not subject to endless special, undocumented fragility.
cmake, just, conan, meson, and bazel (and forks) exist.
But I've still yet to see a proper build system that does feature detection in parallel and concurrently, or supports live, incremental, continuous, cached rebuilding.
That's an improvement, but ultimately not enough, actually. The article hits at this, and you should definitely read it:
Being able to send xz patches to the Linux kernel would have been a nice point of leverage for Jia Tan's future work. We're not at trusting trust [1] levels yet, but it would be one step closer.
I think one of the good things to come out of this may be an increased sense of conservatism around upgrading. Far too many people, including developers, seem to just accept upgrades as always-good instead of carefully considering the risks and benefits. Raising the bar for accepting changes can also reduce the churn that makes so much software unstable.
> Far too many people, including developers, seem to just accept upgrades as always-good instead of carefully considering the risks and benefits.
Another example of this was log4j: if you were still using the old 1.x log4j versions, you wouldn't have been vulnerable to the log4shell vulnerability, since it was introduced early in the 2.x series. The old 1.x log4j versions had other known vulnerabilities, but only if you were using less common appenders or an uncommon server mode or a built-in GUI log viewer (!); the most common use of log4j (logging into a local file) was not exposed to any of these, and in fact, you could remove the vulnerable classes and still have a functional log4j setup (see for instance https://www.petefreitag.com/blog/log4j-1x-mitigation/ which I just found on a quick web search).
Did log4shell (and a later vulnerability which could only be exploited if you were using Java 9 or later, because it depended on a new method which was introduced on Java 9) lead people to question whether always being on the "latest and greatest" was a good thing? No, AFAIK the opposite happened: people started to push even harder to keep everything on the latest release, "so that when another vulnerability happens, upgrading to a fixed version (which is assumed to be based on the latest release) will be easy".
> Another example of this was log4j: if you were still using the old 1.x log4j versions, you wouldn't have been vulnerable to the log4shell vulnerability
Lol, this exact thing happened at my last gig. When I first learned of the vulnerability I panicked, until I found out we were so outdated it didn't affect us. We had a sad laugh about it.
> "so that when another vulnerability happens, upgrading to a fixed version (which is assumed to be based on the latest release) will be easy".
I think there is some truth to this motivation though - if you are on an ancient 1.X version and have to jump a major version of two, that almost always causes pain depending on how critical the service or library is. I don't pretend to know the right answer but I always tend to wait several versions before upgrading so any vulnerabilities or fixes can come by the time I get to the upgrade.
> "so that when another vulnerability happens, upgrading to a fixed version (which is assumed to be based on the latest release) will be easy".
Isn't this still true, generally speaking? How many servers are daily compromised due to out of date, exploitable versions of software versus XZ-like sophisticated attack?
About a decade ago, the industry shifted from slow, carefully evaluated infrequent updates (except for security) to frequent, almost daily updates that are mandatory. I'd say this was pioneered by the Chromium team and proved to be beneficial. The rest of the industry followed.
Now we're in a position where most projects update so quickly, that you don't really have a choice. If you need to update one component, there's a good chance it will require that many more as dependencies in your project will require an update to be compatible.
The industry as a whole sacrificed stability and some aspects of security for faster advancements and features. Overall I'd say the net benefit is positive, but its times like these that remind us that perhaps we need to slow things down just a little and do a bit of a course correction to bring things into a better balance.
A newly-introduced security issue tends to have very limited exploitability, because it's valuable, not-yet well understood, and public exploits are yet to be developed.
Compare to that a similar vulnerability in an older package: chances are that everything about it has been learned and is publicly known. Exploits have become a commodity and are now part of every offensive security distro on the planet. If you run that vulnerable version, there's a real risk that a non-targeted campaign will randomly bite you.
and when you do get a security issue and you're using a 10 year old version the upgrade is going to be really really difficult vs incremental upgrades when they are available. or are you going to fork and assume responsibility for that library code too?
This drove me crazy at previous companies where our build system dependencies were specified with wildcards so they would just auto pull in the latest version whenever you built the project. Not only are there security issues with that, as seen here, but it destroys your ability to have deterministic builds; every time you build the project it might be building against a different set of dependencies. You lose the ability to even compare builds against each other, because even though the builds are for the same exact project code, the dependency binaries might be different, and so the two builds could run differently.
The 2017 WannaCry ransomware attack would be a good counter example, the virus spread even though it was already fixed in an update from MS a bit more than a month before.
I'm kinda the opposite. Way too many times I've seen "upgrades" actively remove things I liked and add things I hate. I hold off on letting mobile apps update because they almost always get worse, not better.
I think it was Ted Neward who argued that libraries should come with a core and ancillary packages. Only the stuff really necessary and broadly popular goes into the base library, and all of the low use features should go into one or more ancillary libraries.
I still think this is the way to go, but it does move some of the problem around in that you have to pick good cleave lines such that people aren’t forced to pull 5 libraries in for 5 features they need. You want one software domain to use two or three and another to use an overlapping set of three.
Wouldn't that just result in exploits written for old versions? A successful exploit for something that everyone is running might be worse, than a backdoor on blending edge systems.
Everyone being on different versions results in something like a moving target
well, it's the bazaar vs the cathedral, isn't it? bazaar moves a lot faster. Everyone likes that part, except when it breaks things, and when they have to chase an upstream that's constantly churning, etc. but most people don't consider that a cathedral itself might have some engineering merit too. cathedrals are beautiful and polished and stable.
I highly encourage people to try freeBSD sometime. Give ports a try (although the modern sense is that poudrie is better even if you want custom-built packages). See how nicely everything works. All the system options you need go into rc.conf (almost uniformly). Everything is documented and you can basically operate the system out of the FreeBSD Handbook documentation (it's not at all comparable to the "how to use a window or a menu" level intro stuff the linux provides). You can't do that when everything is furiously churning every release. everything just works, everything is just documented, it's an experience when you're coming from linux.
and that forum post from 2007 on how to tweak a service script is probably still valid, because BSD hasn't had 3 different init systems over that timespan etc.
just like "engineering is knowing how to build a bridge that barely doesn't fall over", engineering here is knowing what not to churn, and fitting your own work and functionality extensions into the existing patterns etc. like it doesn't have to be even "don't make a bigger change than you have to", you just have to present a stable userland and stable kernel interface and stable init/services interface. the fact that linux doesn't present a stable kernel interface is actually fairly sketchy/poor engineering, it doesn't have to be that way, a large subset of kernel interfaces probably should be stable.
I don't believe this will result in conservatism around upgrading nor that it would be good if it did. The vast, vast majority of security vulnerabilities are due to regular bugs and once the details are known, they can be exploited. Not patching leaves you open to these exploits.
Supply chain attacks are a real threat too, but being conservative about upgrading to improve your security posture is like saying "I'm worried someone with the key to my house might rob me, so I'm going to remove the lock from my door."
Maybe one of the outcomes of this could be a culture change in FOSS towards systematically banning rude consumers in Github issues, or, just in general, a heightened community awareness making us coming down on them way harder when we see it happen.
The attackers will leverage any culture that helps them accomplish their goals.
If being rude and pushy doesn’t work, the next round will be kind and helpful. Don’t read too much into the cultural techniques used, because the cultural techniques will mirror the culture at the time.
The Jia Tan character was never rude. If you make rudeness the thing that throws a red flag, then ‘nice’ fake accounts will bubble up to do the pressuring.
The assumption is that the group behind this attack had sock puppets that were rude to Lasse Collin, to wear him down, and then Jia Tan swept in as the savior.
Jia Tan wasn't rude, but the original maintainer Laser Collin probably wouldn't have been as burned out and willing to give responsibility to them if the community wasn't as rude and demanding of someone doing free work for them.
I think we need to start paying more of these open source maintainers and have some staff/volunteers that can help them manage their git hub issue volume.
I want to caution against taking a good thing too far.
There's a certain kind of talented person who is all too conscious of their abilities and is arrogant, irascible, and demanding as a result. Linus Torvalds, Steve Jobs, Casey Muratori come to mind. Much as we might want these characters to be kinder, their irascibility is inseparable from their more admirable qualities.
Sometimes good things, even the best things, are made by difficult people, and we would lose a lot by making a community that alienates them.
That's a tough one - It's hard to fully disagree but in my experience you can have all the benefits without the poison. Accepting the poison just because of the benefits is kind of just giving up. I don't feel like the your hypothesis that the two are irrevocably linked holds up under examination.
Linus Torvalds is apparently trying to do better (although I haven't followed up with the progress), but more importantly, while he might be (have been) unnecessarily rude and aggressive, he's not entitled (as far as I know). I don't think he would jump into an issue tracker of some project he doesn't maintain and demand that certain changes be made.
While I don't condone some of the treatment he received, that situation was extremely different.
A user reported a safety issue, the maintainer said it was safe. Then it was proven that it was in fact unsafe, and the maintainer justified it with performance. Then a PR was filed which was safe and did not regress performance, and the maintainer rejected it with "this patch is boring"
The behavior of both sides was deeply unacceptable. If someone identifies a legitimate issue and files a PR to fix it, don't insult them by calling the patch "boring" and don't reject it solely on that basis.
I don't want to excuse rudeness or a sense of entitlement. But I think we can still understand where it comes from. A lot of these people probably work on crappy codebases where "let's just add a random dependency without any vetting" was the norm, they might have to deal with production issues etc. There's probably a systemic issue behind it, that our industry relies too much on unpaid labour and is usually not willing to contribute back.[0]
[0] Funnily enough, just a week or two ago, I fixed an issue in an OS project that we introduced at work. It was an easy frontend fix even for someone like me who doesn't do frontend and barely knows how to spell Vue. And more importantly, in the issue description somebody already wrote exactly what causes the bug and what would need to change - the only thing left was finding the place where to make the (one-line) change. Somehow that issue had been open for 2 years but nobody of the several people who complained (nor the maintainer) had bothered to fix it. After I made a PR, it was merged within a day.
Being rude is... unimportant. A lot of people think being passive aggressive is being polite when it's actually being rude + deceitful. There's nothing wrong with being direct, which some mistake for rude. I find it refreshing.
The issue here is the attackers will quickly move away from an individual attacking you to the group attacking you. The person writing the infected code will never be a jerk to you at all. You'll just suddenly see a huge portion of your mailing list move against you ever so slightly.
We've complained about bots in social media for a long time, but how many people in open source discussions are shady manipulative entities?
But in this case he was getting hit by both someone willing to help and then multiple people complaining that things were taking too long. And when you yourself feel like things are taking too long then you’re probably more susceptible to all this.
I did an engineering/program manager role for 8 years and people pretty much always did what I asked if I showed up at their desk or bothered their boss.
"Squeaky wheel gets the grease?"
But I too like to think that I prioritize my children on merit rather than fuss level. For some reason they continue to cry despite me claiming I don't react to it.
True, but a determined adversary like JiaTan/Jugar has an ace up their sleeve: they are good enough, and patient enough, to be able to fork the base project, spend a year or two making it better than the original (releasing the head of steam built up from legitimate requests that the old, overworked maintainer never got too, building goodwill in the process) and then convincing the distros to pick up their fork instead of the older original. At which point it really is game over.
The social side of this is really haunting me over the last days. It's surprisingly easy to pressure people to giving up control. I've been there myself. I can't even imagine how devastating this must be to the original author of XZ, especially if he is dealing with other personal issues as well. I hope at least this will serve a strong example to other open source people, to never allow others to pressure them into something they are not comfortable with.
It makes Rich Hickey’s „Open Source Is Not About You” [0] particularly poignant.
As a hobbyist developer/maintainer of open source projects, I strive to remember that this is my gift to the world, and it comes with no strings attached. If people have any expectations about the software, it’s for them to manage; if they depend on it somehow, it’s their responsibility to ensure timely resolution of issues. None of this translates to obligations on my part, unless I explicitly make promises.
I empathize with Lasse having been slowed down by mental issues. I have, too. And we need to take good care of ourselves, and proactively prevent the burden of maintainership from exacerbating those issues.
Anyone and everyone in the OSS world should be concerned about this too. You have nation state level actors out there with massive amounts of information on you.
How much information have you leaked to data brokers? These groups will know how much debt you're in. The status of your relationships. Your health conditions and medications? It would not take much on their part to make your life worse and increase your stress levels. Just imagine things like fake calls from your bank saying that debt of yours has been put in collections.
This is why I find some disclaimers in some open source projects quite superfiscial, that the software is provided as is without any warranty. Of course it is, this should be the obvious default.
If there is a law that would entitle a user to more, it is a bug in legislation that needs urgent fixing.
I see this as sort of the pivot on how people choose an open source license. When you feel like you are building the thing for others use a gplish license, it has all sorts of clauses around getting everyone to play nice. Building the thing for yourself however, I think the bsd style license makes more sense. you don't really care what anyone else is doing with it, you don't want to form a community. however, because it is trivial to share source code, you do so.
Look how brilliantly they selected their target project:
(1) xz and the lib are widely used in the wild including linux kernel, systemd, openSSH; (2) single maintainer, low rate of maintenance; (3) the original maintainer has other problems in his life distracting them from paying closer attention to the project.
I am wondering how many other OSS projects look similar and can be targeted in similar ways?
I'm thinking 95% of home automation which is full of obscure devices and half baked solutions which get patched up by enthusiasts and promptly forgotten about.
Controlling someone's lights is probably less important than Debian's build fleet but it's a scary proposition for the impacted individual who happens to use one of those long tail home assistant integrations or whatever.
A takeaway for me is to be extremely tight with personal information on the internet. People will use this to craft a situation to fool you.
Are you married? Have a house? Pets? Children? Sick parent? Gay? Trans? Mental health issues? Disabled? All of this can be used against you. Be careful where and how you share stuff like this. I know it's not "cool" to be mysterious online anymore, but it creates a much larger attack surface. People can still engage with groups around these things, but better to do it with various personas than to have one trackable identity with everything attached to it.
We're in a tech slowdown right now. There are people who got used to a certain lifestyle who now have "seeking work" on their LinkedIn profiles, and who have property taxes in arrears that are listed in county newspapers-of-record. If you're an intelligence operative in the Silicon Valley area, these guys should be easy pickings. An envelope full of cash to make some financial problems go away in exchange for a few commits on the FOSS projects they contribute to or maintain.
The Jigar Kumar nudges are so incredibly rude. I would have banned the account, but perhaps they contributed something positive as well that isn't mentioned.
I wonder if it would be possible to crowdsource FOSS mailing list moderation.
There is a good chance that everyone in that thread except the original maintainer is in on the act. It's likely that all those accounts are managed by a single person or group. Targeting just one account for rudeness isn't going to help, if that's true.
Yeah, same. We should be much more willing to kick jerks out of our work spaces. The work is hard enough as it is without also being shit on while you do it.
Yea people are too accepting of allowing asshats like the Jigar messages.
Simple ban and get the fuck out. Too often I've dealt with people trying to rationalize it as much as "o its just cultural, they don't understand". No, get the fuck out.
But hey I'm a NYer and telling people to fuck off is a past time.
Not surprising, unfortunately. You'd think malicious actors would be nice to people they're trying to deceive. But after watching a few Kitboga videos, I learned that they more often yell, abuse, and swear at their victims instead.
>I wonder if it would be possible to crowdsource FOSS mailing list moderation.
I think this could be a genuine use of an AI: to go through all of the shit, and have it summarized in a fashion that the user wants: distant and objective, friendly, etc. It could provide an assessment on the general tone, aggregate the differently phrased requests, many things like that.
Crowdsourcing would works best with the reddit / hacker news model I feel, where discussion happens in tree styled threads, and users can react to messages in ways that are not text, but something meta, like a vote or a reaction indicating tone.
Both of these have significant downsides, but significant upsides too. People pick the mailing list in a similar way.
It's time for more of the big vendors who use these projects in their offerings to step up and give people running these small projects more resources and structure. $20k to have maintainers for each project actually meet twice a year at a conference is chump change for the biggest vendors, especially when compared against the cost of the audits they'll now be doing on everything Jia Tan and Co. touched.
As an OSS maintainer, $20k wouldn't help me enough unless I was retired. The issue is not money (or not just money), but time. If a maintainer has a full-time job, they may not have time, and developers/maintainers tend to have full-time jobs, so...
Now maybe one could build a career out of OSS maintainerships, with work/time funded by lots of donations much smaller than a salary but amounting to a salary.
I’ve given semi-popular projects that I no longer had the bandwidth to maintain to random people who bothered to email, no pressuring needed. While those projects are probably four to five magnitudes less important than xz, still thousands of people would be affected if the random dude who emailed was malicious. What should I have done? Let the projects languish? Guess I’ll still take the chance in the future.
If it's open source they can just fork it, and if you're no longer maintain yours you can put a link over to their fork. (Or any other active forks). It's still on the user to vet new forms.
> What should I have done? Let the projects languish?
Yes, if you can't find a successor you trust then let someone fork the project and build trust from 0 rather than transferring trust others' trust in you by handing over the project. This doesn't just apply to security concerns btw. - plenty of other ways in which a new maintainer might end up making the project worse (intentionally or through incompetence) compared to it not receiving any updates.
In "Ghost in the Wires" Kevin Mitnik details one of the ways he obtained information was via a law enforcement receptionist* who he managed to trick into believing he was law enforcement over the phone. He obtained information this way multiple times over multiple years, and fostered a phone based friendship with this woman. He seemed to have no qualms in doing this.
He was also turned on by multiple people who he considered close friends. In the book it did not seem that he had considered that it might not be a "them" problem.
*my details may be off here, I read it some time ago
Your experiences may differ, but I'd say pretty much anyone who lived through the past few years has reason enough to pay careful attention to their mental health.
> merges hidden backdoor binary code well hidden inside some binary test input files. [...] Many of the files have been created by hand with a hex editor, thus there is no better "source code" than the files themselves.
So much for the folks advocating for binary (driver) blobs in OSS t support otherwise unsupported hardware.
It's either in source form and reproducable or it's not there.
>It's either in source form and reproducable or it's not there.
Wanna know how I know you haven't read into the discussion much?
There are a whole lot of binary test cases in software. Especially when you're dealing with things like file formats and test cases that should specifically fail on bad data of particular types.
> There are a whole lot of binary test cases in software.
That's not how I read GP's point. If even binary blobs in test cases are a place where backdoors are, now as a matter of fact, hidden then, certainly, among the folks advocating for binary drivers in FOSS, there are some who are already --or planning to-- add backdoors there.
Binary blobs are all terrible, terrible, terrible ideas.
Builds should be 100% reproducible from source, bit for bit. At this point it's not open up for discussion anymore.
Readit News
One thing that could be added, for the sake of completeness: in the part "Attack begins", toward the end, when they are pushing for updating xz in the major distros, Ubuntu and Debian are mentioned but not Fedora.
Looks like the social engineering/pressuring for Fedora started at least weeks before 2024 March 04, according to a comment by @rwmj on HN [1]. I also found this thread on Fedora's devel list [2], but didn't dig too much.
[1] https://news.ycombinator.com/item?id=39866275
[2] https://lists.fedoraproject.org/archives/list/devel@lists.fe...
He notes that while he was unsatisfied with some of the changes Tan introduced, Tan was nonetheless extremely helpful.
Deleted Comment
https://lists.fedoraproject.org/archives/list/devel@lists.fe... (https://archive.ph/diGNB)
This is the rough sequence of events in Fedora:
https://lists.fedoraproject.org/archives/list/devel@lists.fe... (https://archive.ph/e0SdX)
Its time to stop accepting that the way we've done this in the past is the way we will continue doing it ad infinitum.
There might have been a time where it worked, but people started to use plugins for all kinds of reasons, quite good ones in most cases. Findbugs, code coverage, source code generation,...
Today, a maven project without plugins is rare. Maven brought us 95%, but there is a long tail left to cover.
> 99.9% of all projects have the capability to be built with trivial declarative rulesets.
Only if you forbid bootstrapping, which all projects ultimately rely on at some point in their supply chain.
I would like to see this on the language level, not just on the OS level.
Sounds like it would be impossible but maybe my thinking is just enclosed and not free.
Who will pay for all the rewriting you want done? Or even just for the new frameworks that are "scrutable"? How do we guarantee that the result is not inscrutable to you or others?
There is so much knee-jerking in this xz debacle.
(And I say this / ask these questions with no love for autoconf/m4/sh.)
We should be moving from No Obvious Bugs to Obviously No Bugs (Tony Hoare).
That’s the beauty of FOSS: if you don’t like what’s written, you can write your own replacement.
Reading the timeline, the root cause is pure social engineering. The technical details could be swapped out with anything. Sure there are aspects unique to xz that were exploited such as the test directory full of binaries, but that's just because they happened to be the easiest target to implement their changes in an obfuscated way, that misses the point - once an attacker has gained maintainer-ship you are basically screwed - because they will find a way to insert something, somehow, eventually, no matter how many easy targets for obfuscation are removed.
Is the real problem here in handing over substantial trust to an anonymous contributor? If a person has more to lose than maintainership then would this have happened?
That someone can weave their way into a project under a pseudonym and eventually gain maintainership without ever risking their real reputation seems to set up quite a risk free target for attackers.
Unless there's some practical way of comprehensively solving The Real Problem Here, it makes a lot of sense to consider all reasonable mitigations, technical, social or otherwise.
> If a person has more to lose than maintainership then would this have happened?
I guess that's one possible mitigation, but what exactly would that look like in practice? Good luck getting open source contributors to accept any kind of liability. Except for the JiaTans of the world, who will be the first to accept it since they will have already planned to be untouchable.
It could also be called an industry problem, where we rely on other's code without running proper checks. This seems to be an emerging realization, with services like socket.dev starting to emerge.
Build infrastructure should be minimal, standardized, and not subject to endless special, undocumented fragility.
cmake, just, conan, meson, and bazel (and forks) exist.
But I've still yet to see a proper build system that does feature detection in parallel and concurrently, or supports live, incremental, continuous, cached rebuilding.
And all code that gets linked into security critical applications/libraries needs to be covered by under some sort of security focused code review.
So no patching the compression code that openSSL links to with random junk distribution maintainers.
Being able to send xz patches to the Linux kernel would have been a nice point of leverage for Jia Tan's future work. We're not at trusting trust [1] levels yet, but it would be one step closer.
[1]: https://research.swtch.com/nih
Dead Comment
Another example of this was log4j: if you were still using the old 1.x log4j versions, you wouldn't have been vulnerable to the log4shell vulnerability, since it was introduced early in the 2.x series. The old 1.x log4j versions had other known vulnerabilities, but only if you were using less common appenders or an uncommon server mode or a built-in GUI log viewer (!); the most common use of log4j (logging into a local file) was not exposed to any of these, and in fact, you could remove the vulnerable classes and still have a functional log4j setup (see for instance https://www.petefreitag.com/blog/log4j-1x-mitigation/ which I just found on a quick web search).
Did log4shell (and a later vulnerability which could only be exploited if you were using Java 9 or later, because it depended on a new method which was introduced on Java 9) lead people to question whether always being on the "latest and greatest" was a good thing? No, AFAIK the opposite happened: people started to push even harder to keep everything on the latest release, "so that when another vulnerability happens, upgrading to a fixed version (which is assumed to be based on the latest release) will be easy".
Lol, this exact thing happened at my last gig. When I first learned of the vulnerability I panicked, until I found out we were so outdated it didn't affect us. We had a sad laugh about it.
> "so that when another vulnerability happens, upgrading to a fixed version (which is assumed to be based on the latest release) will be easy".
I think there is some truth to this motivation though - if you are on an ancient 1.X version and have to jump a major version of two, that almost always causes pain depending on how critical the service or library is. I don't pretend to know the right answer but I always tend to wait several versions before upgrading so any vulnerabilities or fixes can come by the time I get to the upgrade.
Isn't this still true, generally speaking? How many servers are daily compromised due to out of date, exploitable versions of software versus XZ-like sophisticated attack?
Now we're in a position where most projects update so quickly, that you don't really have a choice. If you need to update one component, there's a good chance it will require that many more as dependencies in your project will require an update to be compatible.
The industry as a whole sacrificed stability and some aspects of security for faster advancements and features. Overall I'd say the net benefit is positive, but its times like these that remind us that perhaps we need to slow things down just a little and do a bit of a course correction to bring things into a better balance.
How many security issues spring from outdated packages vs packages updated too hastily?
A newly-introduced security issue tends to have very limited exploitability, because it's valuable, not-yet well understood, and public exploits are yet to be developed.
Compare to that a similar vulnerability in an older package: chances are that everything about it has been learned and is publicly known. Exploits have become a commodity and are now part of every offensive security distro on the planet. If you run that vulnerable version, there's a real risk that a non-targeted campaign will randomly bite you.
I still think this is the way to go, but it does move some of the problem around in that you have to pick good cleave lines such that people aren’t forced to pull 5 libraries in for 5 features they need. You want one software domain to use two or three and another to use an overlapping set of three.
Everyone being on different versions results in something like a moving target
I highly encourage people to try freeBSD sometime. Give ports a try (although the modern sense is that poudrie is better even if you want custom-built packages). See how nicely everything works. All the system options you need go into rc.conf (almost uniformly). Everything is documented and you can basically operate the system out of the FreeBSD Handbook documentation (it's not at all comparable to the "how to use a window or a menu" level intro stuff the linux provides). You can't do that when everything is furiously churning every release. everything just works, everything is just documented, it's an experience when you're coming from linux.
and that forum post from 2007 on how to tweak a service script is probably still valid, because BSD hasn't had 3 different init systems over that timespan etc.
just like "engineering is knowing how to build a bridge that barely doesn't fall over", engineering here is knowing what not to churn, and fitting your own work and functionality extensions into the existing patterns etc. like it doesn't have to be even "don't make a bigger change than you have to", you just have to present a stable userland and stable kernel interface and stable init/services interface. the fact that linux doesn't present a stable kernel interface is actually fairly sketchy/poor engineering, it doesn't have to be that way, a large subset of kernel interfaces probably should be stable.
Supply chain attacks are a real threat too, but being conservative about upgrading to improve your security posture is like saying "I'm worried someone with the key to my house might rob me, so I'm going to remove the lock from my door."
If being rude and pushy doesn’t work, the next round will be kind and helpful. Don’t read too much into the cultural techniques used, because the cultural techniques will mirror the culture at the time.
I think we need to start paying more of these open source maintainers and have some staff/volunteers that can help them manage their git hub issue volume.
If they want things done quickly they can do it themselves
There's a certain kind of talented person who is all too conscious of their abilities and is arrogant, irascible, and demanding as a result. Linus Torvalds, Steve Jobs, Casey Muratori come to mind. Much as we might want these characters to be kinder, their irascibility is inseparable from their more admirable qualities.
Sometimes good things, even the best things, are made by difficult people, and we would lose a lot by making a community that alienates them.
A user reported a safety issue, the maintainer said it was safe. Then it was proven that it was in fact unsafe, and the maintainer justified it with performance. Then a PR was filed which was safe and did not regress performance, and the maintainer rejected it with "this patch is boring"
The behavior of both sides was deeply unacceptable. If someone identifies a legitimate issue and files a PR to fix it, don't insult them by calling the patch "boring" and don't reject it solely on that basis.
[0] Funnily enough, just a week or two ago, I fixed an issue in an OS project that we introduced at work. It was an easy frontend fix even for someone like me who doesn't do frontend and barely knows how to spell Vue. And more importantly, in the issue description somebody already wrote exactly what causes the bug and what would need to change - the only thing left was finding the place where to make the (one-line) change. Somehow that issue had been open for 2 years but nobody of the several people who complained (nor the maintainer) had bothered to fix it. After I made a PR, it was merged within a day.
Deleted Comment
We've complained about bots in social media for a long time, but how many people in open source discussions are shady manipulative entities?
I did an engineering/program manager role for 8 years and people pretty much always did what I asked if I showed up at their desk or bothered their boss.
"Squeaky wheel gets the grease?"
But I too like to think that I prioritize my children on merit rather than fuss level. For some reason they continue to cry despite me claiming I don't react to it.
Given the current situation, I'm slightly worried about Fedora's planned move to zlib-ng instead of zlib in the next release (https://fedoraproject.org/wiki/Changes/ZlibNGTransition).
As a hobbyist developer/maintainer of open source projects, I strive to remember that this is my gift to the world, and it comes with no strings attached. If people have any expectations about the software, it’s for them to manage; if they depend on it somehow, it’s their responsibility to ensure timely resolution of issues. None of this translates to obligations on my part, unless I explicitly make promises.
I empathize with Lasse having been slowed down by mental issues. I have, too. And we need to take good care of ourselves, and proactively prevent the burden of maintainership from exacerbating those issues.
[0]: https://gist.github.com/g1eny0ung/9e7d4d0f72547a8d156452e76f...
Anyone and everyone in the OSS world should be concerned about this too. You have nation state level actors out there with massive amounts of information on you. How much information have you leaked to data brokers? These groups will know how much debt you're in. The status of your relationships. Your health conditions and medications? It would not take much on their part to make your life worse and increase your stress levels. Just imagine things like fake calls from your bank saying that debt of yours has been put in collections.
If there is a law that would entitle a user to more, it is a bug in legislation that needs urgent fixing.
(1) xz and the lib are widely used in the wild including linux kernel, systemd, openSSH; (2) single maintainer, low rate of maintenance; (3) the original maintainer has other problems in his life distracting them from paying closer attention to the project.
I am wondering how many other OSS projects look similar and can be targeted in similar ways?
Controlling someone's lights is probably less important than Debian's build fleet but it's a scary proposition for the impacted individual who happens to use one of those long tail home assistant integrations or whatever.
Are you married? Have a house? Pets? Children? Sick parent? Gay? Trans? Mental health issues? Disabled? All of this can be used against you. Be careful where and how you share stuff like this. I know it's not "cool" to be mysterious online anymore, but it creates a much larger attack surface. People can still engage with groups around these things, but better to do it with various personas than to have one trackable identity with everything attached to it.
We're in a tech slowdown right now. There are people who got used to a certain lifestyle who now have "seeking work" on their LinkedIn profiles, and who have property taxes in arrears that are listed in county newspapers-of-record. If you're an intelligence operative in the Silicon Valley area, these guys should be easy pickings. An envelope full of cash to make some financial problems go away in exchange for a few commits on the FOSS projects they contribute to or maintain.
I wonder if it would be possible to crowdsource FOSS mailing list moderation.
Messages like Jigar’s are kind of par for the course.
Yeah, same. We should be much more willing to kick jerks out of our work spaces. The work is hard enough as it is without also being shit on while you do it.
Simple ban and get the fuck out. Too often I've dealt with people trying to rationalize it as much as "o its just cultural, they don't understand". No, get the fuck out.
But hey I'm a NYer and telling people to fuck off is a past time.
I think this could be a genuine use of an AI: to go through all of the shit, and have it summarized in a fashion that the user wants: distant and objective, friendly, etc. It could provide an assessment on the general tone, aggregate the differently phrased requests, many things like that.
Crowdsourcing would works best with the reddit / hacker news model I feel, where discussion happens in tree styled threads, and users can react to messages in ways that are not text, but something meta, like a vote or a reaction indicating tone.
Both of these have significant downsides, but significant upsides too. People pick the mailing list in a similar way.
It's time for more of the big vendors who use these projects in their offerings to step up and give people running these small projects more resources and structure. $20k to have maintainers for each project actually meet twice a year at a conference is chump change for the biggest vendors, especially when compared against the cost of the audits they'll now be doing on everything Jia Tan and Co. touched.
Now maybe one could build a career out of OSS maintainerships, with work/time funded by lots of donations much smaller than a salary but amounting to a salary.
Yes, if you can't find a successor you trust then let someone fork the project and build trust from 0 rather than transferring trust others' trust in you by handing over the project. This doesn't just apply to security concerns btw. - plenty of other ways in which a new maintainer might end up making the project worse (intentionally or through incompetence) compared to it not receiving any updates.
Put a link saying "Hey this guy forked my project, I won't maintain it anymore, he may add malware, review and use at your own risk"
https://en.wikipedia.org/wiki/The_Strange_Case_of_Mr_Pelham
He was also turned on by multiple people who he considered close friends. In the book it did not seem that he had considered that it might not be a "them" problem.
*my details may be off here, I read it some time ago
In another thought, I hope the nukes aren’t on a rolling Debian distro.
Deleted Comment
So much for the folks advocating for binary (driver) blobs in OSS t support otherwise unsupported hardware.
It's either in source form and reproducable or it's not there.
Wanna know how I know you haven't read into the discussion much?
There are a whole lot of binary test cases in software. Especially when you're dealing with things like file formats and test cases that should specifically fail on bad data of particular types.
That's not how I read GP's point. If even binary blobs in test cases are a place where backdoors are, now as a matter of fact, hidden then, certainly, among the folks advocating for binary drivers in FOSS, there are some who are already --or planning to-- add backdoors there.
Binary blobs are all terrible, terrible, terrible ideas.
Builds should be 100% reproducible from source, bit for bit. At this point it's not open up for discussion anymore.
Forbidding binary data in source repositories is neither practical nor solving the issue.
Binary code can be hidden steganograhically in e. g. image files or text files.