A title that actually describes the post, mostly paraphrasing the first paragraph:
Reasons why this buffer overflow wasn't caught earlier despite doing all the right things
And then to give those reasons:
- "each component is fuzzed independently" ... "This fuzzer might have produced a SECKEYPublicKey that could have reached the vulnerable code, but as the result was never used to verify a signature, the bug could never be discovered."
- "There is an arbitrary limit of 10000 bytes placed on fuzzed input. There is no such limit within NSS; many structures can exceed this size. This vulnerability demonstrates that errors happen at extremes"
- "combined [fuzzer] coverage metrics [...]. This data proved misleading, as the vulnerable code is fuzzed extensively but by fuzzers that could not possibly generate a relevant input."
The conclusion is, of course, to fix those problems if your code base also has them, but also "even extremely well-maintained C/C++ can have fatal, trivial mistakes".
The whole post is a giant blinking red sign that says (or should say) "Fuzzing is a horribly ineffective workaround for a treacherous language."
No offense to the many bright and capable people who have worked hard on the C/C++ language, tools, compilers, libraries, kernels, etc over the years, but we will someday look back on it as asbestos and wonder why we kept at it for so damn long.
No issue with the first sentence of your message at all, but...
> No offense to the many bright and capable people who have worked hard on the C/C++ language, tools, compilers, libraries, kernels, etc over the years, but we will someday look back on it as asbestos and wonder why we kept at it for so damn long.
We won't wonder at all. We will understand that those people are the ONLY ones that stepped up to the task over 50 years to write this kind of software, organize standards bodies for their languages and platforms, get their software packaged as part of mainstream operating systems and out into the world, deal with patches from users, and help millions of other people make a living, enable the internet to happen, etc.
We will wonder why with all the millions of lines of C/C++ reference code available to be perused and then rewritten in Rust, Pascal, C#, Zig or Nim, and the vociferousness of their advocates, why that didn't happen in a reasonable timeframe.
We will wonder why all the Lisp and Haskell programmers who sneer down their nose at working C/C++ programmers in forums like this on a daily basis, didn't get off their asses with their One True Language (TM) and come to the worlds rescue.
The answer will be: these people aren't doers - they are talkers. It's one thing to get a toy proof of concept OS in your favourite language of choice that supports like 5 drivers. It's another thing to contribute to and build an ecosystem depended on by millions of developers daily. C/C++ people may not always be the sharpest tools in the shed, and they may be a dime a dozen. But they know how to organize themselves in loose groups of more than just a few developers, and work with other people.
We may have issues with the quality of what they ship on a frequent basis, but at least they ship instead of posting endless comments on internet forums about how it's all the others peoples fault.
More strongly: the idea that "fuzzing" is "doing all the right things" is insane and disappointing of a narrative. The code snippets I am seeing here are ridiculous with manual, flag-based error handling? There is a use of something like memcpy that isn't hidden behind some range abstraction that manages all of the bounds?! This code probably is riddled with bugs, because they are doing all the wrong things, and fuzzing the result counts for nothing. C++ sucks, but it is way better than this.
> but we will someday look back on it as asbestos and wonder why we kept at it for so damn long.
Maybe. But there is good reason for C dominance - it's low level and close to systems as in "syscalls and stuff". And we need that level of systems control, not only for max performance but also for not loosing what is available in hw and os.
Asm is the other option to have full functionality :) Maybe it's just case of available libraries but still C is curently only one option for feature completness. And on low levels all you have is bytes and bits and calls and that allows to do everything with bits, bytes and calls - Core Wars style - and most of languages try to prevent or easy use of that by cutting possibilities, eg. "goto is so nono !11".
And yes, C is not perfect and bullshit even grows, eg. in last years NULL checks are "optimized-out"...
C actually should be improved and not fucked up into forced disuse...
Is it OK to remove modern C++ from your statement ? Using a `std::vector<std::byte>` wouldn't cause this problem. Don't know why everyone always berates C++ for vulnerabilities in traditional C code.
Unfortunately it's too much work to throw away all code written in unsafe languages. So it's valuable to try and improve tools and techniques that make these languages less unsafe.
What's special here is the bug is a memory corruption, and memory corruption bugs in such libraries are usually instantly security bugs.
Otherwise, the same story could be told as a generic software testing joke: "unit-tests are short-sighted and coverage lies", i.e. an "extremely well-maintained codebase, with extensive unittest, >98% test coverage and constantly scanned by all-static-analyzers-you-may-come-up" can have fatal, trivial bugs.
Ah, brings to mind one of my favorite Dijsktra quotes, "Program testing can be used to show the presence of bugs, but never to show their absence!"
I've never understood that to mean that he wasn't in favor of automated testing, only that it's got its limits. In this case, they now know a test case that was missing.
> What's special here is the bug is a memory corruption, and memory corruption bugs in such libraries are usually instantly security bugs.
Is that special? Are there buffer overflow bugs that are not security bugs? It could be just my bubble as a security consultant, since (to me) "buffer overflow" assumes remote code execution is a given. It's not my area of expertise, though, so perhaps indeed not all reachable buffer overflows are security issues. (Presuming no mitigations, of course, since those are separate from the bug itself.)
> - "There is an arbitrary limit of 10000 bytes placed on fuzzed input. There is no such limit within NSS; many structures can exceed this size. This vulnerability demonstrates that errors happen at extremes"
This is the one that seemed short sighted to me. It's a completely arbitrary (and small!) limit that blinded the fuzzer to this very modest sized buffer overflow.
The buffer holds 2K, so this limit alone which exceeds the buffer by 8K'ish didn't blind the fuzzer. It's not clear a larger input would've caught anything due to other "what went wrong" items, specifically "each component is fuzzed independently."
The problem is that the search space grows (exponentially?) as you increase the fuzzer’s limit. So there’s a cost, and likely diminishing returns, to raising that limit.
How do you know that it was actually "extremely well-maintained"? Everybody thought OpenSSL was well-maintained since it was used as critical infrastructure by multi-billion dollar megacorporations, but it was actually maintained by two full-time employees and a few part-time volunteers with maybe a quick once-over before a commit if they were lucky. How about sudo, a blindly trusted extremely sensitive program [1], which is maintained by basically one person who has nearly 3,000,000(!) changes[2] over 30 or so years?
Assuming that something is well-maintained because it is important is pure wishful thinking. Absent a specific detailed high quality process, or an audit that they conform to a well-established process that has demonstrably produced objectively high-quality output in a large percentage of audited implementations of that process (thus establishing nearly every instance of the audited process -> high quality output) all evidence indicates that you should assume that these code bases are poorly maintained until proven otherwise[3]. And, even the ones that are demonstrably maintained usually use very low quality processes as demonstrated by the fact that almost nobody working on those projects would be comfortable using their processes on safety-critical systems [4] which is the minimum bar for a high quality process (note the bar is "believe it is okay for safety-critical"). In fact, most would be terrified of the thought and comfort themselves knowing that their systems are not being used in safety-critical systems because they are absolutely not taking adequate precautions, which is a completely reasonable and moral choice of action as they are not designing for those requirements, so it is totally reasonably to use different standards on less important things.
Tavis explains clearly why he thinks it's well maintained in the post, complete with linkages to source code. To paraphrase:
[...] NSS was one of the very first projects included with oss-fuzz [...]
[...] Mozilla has a mature, world-class security team. They pioneered bug bounties, invest in memory safety, fuzzing and test coverage. [... all links to evidence ...]
Did Mozilla have good test coverage for the vulnerable areas? YES.
Did Mozilla/chrome/oss-fuzz have relevant inputs in their fuzz corpus? YES.
Is there a mutator capable of extending ASN1_ITEMs? YES.
I don't think at any point anyone assumed anything.
OpenSSL is a project which is treated as a "somebody else's problem" dependency by everybody, and the extent to which everybody cares about TLS support, it's basically an "open up a TLS socket and what do you mean it's more complicated than that" situation.
By contrast, NSS is maintained by Mozilla as part of Firefox, and, furthermore, its level of concern is deep into the "we don't want to enable certain cipher suites, and we have very exacting certificate validation policies that we are part of the effort in defining"--that is to say, NSS isn't a "somebody else's problem" dependency for Mozilla but a very "Mozilla's problem" dependency.
That said, this is CERT_VerifyCertificate, not mozilla::pkix, and since this is not used in Firefox's implementation of certificate validation, I would expect that this particular code in the library would be less well-maintained than other parts. But the whole library itself wouldn't be in the same camp as OpenSSL.
> Everybody thought OpenSSL was well-maintained since it was used as critical infrastructure by multi-billion dollar megacorporations
I wasn't under the impression anybody who knew the project ever really thought that. Some other people may have assumed that as a default if they hadn't looked into it.
This article spells out a whole bunch of reasoning why this particular library was well maintained though. There's a difference between reasoning based on evidence and assumptions.
I don't know anybody who thought OpenSSL was well-maintained in and before the Heartbleed era (it's a fork of SSLeay, which was Eric Young's personal project). Post-Heartbleed --- a decade ago, longer than the time lapse between SSLay and OpenSSL --- maintenance of OpenSSL has improved dramatically.
I think the main surprising thing here is that people are putting smallish arbitrary limits on the sizes of inputs that they let their fuzzer generate.
With the benefit of a little hindsight, that does feel rather like saying "please try not to find any problems involving overflows".
I agree. I haven’t done a lot of fuzzing, but my understanding is that this is how fuzzing can be helpful. Am I wrong? Or is it more complicated than that?
It's a trade-off. Larger input files may slow the fuzzing process, and therefore explore less of the problem space. You usually want to test many different kinds of inputs, not just more of the same.
OTOH file formats often include sizes of fields, which a fuzzer will set to arbitrarily high values. This tests (some) handling of overly large inputs without files being actually that large.
How realistic is it that this vulnerability can be exploited for $BAD_THINGS?
https://www.mozilla.org/en-US/security/advisories/mfsa2021-5... notes "This vulnerability does NOT impact Mozilla Firefox. However, email clients and PDF viewers that use NSS for signature verification, such as Thunderbird, LibreOffice, Evolution and Evince are believed to be impacted.".
I don’t understand why the “lessons learned” doesn’t recommend always* passing the destination buffer size (using memcpy_s or your own wrapper). It has been a long time since I wrote C++, but when I did this would have been instantly rejected in code review.
*…with, I suppose, potential exceptions in performance-critical code when you control and trust the input; I don’t believe that this code qualifies on either count.
You catch the bug by flagging the use of memcpy instead of something that takes the dest buffer size (like memcpy_s or whatever).
It seems to me linters have been flagging this kind of thing since forever. This code is using a wrapper, "PORT_memcpy", so a default ruleset isn't going to flag it.
So here I guess no one noticed PORT_memcpy == memcpy (or maybe noticed but didn't take the initiative to add a lint rule or deprecation entry or just created an issue to at least port existing code).
Counterexample: msgrcv(). This expects you to not be passing raw buffers, but messages with a particular structure: a long mtype, to specify what type of message it is, and then a char (byte, since this is C) array that is the buffer that contains the rest of the message. You pass these structures to msgsnd() and msgrcv(), along with a size. But the size is the size of the buffer component of the structure, not the size of the structure as a whole. If you pass the size of the structure, it will read sizeof(long) more than your structure can hold. Been bit by that...
So, just passing the size of the destination is something that you can still get wrong, in the case of data more complicated than just a single buffer.
[Edit: You can also design an API to be very misleading, even if it has a length parameter...]
It's not the case here (I think), but this can be common if you move functions around or expose functions that were previously internal.
For example, maybe your function is only called from another one that performs the appropriate bound checks, so checking again becomes redundant. After a simple refactoring, you can end up exposing your function, and screw up.
Usually people say "oh, it's just another typical failure of writing in memory-unsafe C", but here's a slightly different angle: why is this common error is not happening under a single abstraction like "data structure that knows it size"? If C was allowing for such things, then 100000 programs would be using same 5-10 standard structures where the copy-and-overflow bug would be fixed already.
Languages like Rust, of course, provide basic memory safety out of the box, but most importantly they also provide means to package unsafe code under safe API and debug it once and for all. And ecosystem of easy to use packages help reusing good code instead of reinventing your own binary buffers every single damn time, as it's usually done in C.
So maybe it's not the unsafeness itself, but rather inability to build powerful reusable abstractions that plagues C? Everyone has to step on the same rake again and again and again.
But performance! Rust and other languages with bounds checking go out of their way to not do it once it is proven that they don't need to. It would be hard to do that as a data structure.
Well, here comes the type system, so your fancy data structure has zero cost. Rust recently got more support for const generics, so you could encode size bounds right in the types and skip unnecessary checks.
We continue to be reminded that it's hard to write fully memory secure code in a language that is not memory secure?
And by hard, I mean, very hard even for folks with lots of money and time and care (which is rare).
My impression is that Apple's imessage and other stacks also have memory unsafe languages in the api/attack surface, and this has led to remote one click / no click type exploits.
Is there a point at which someone says, hey, if it's very security sensitive write it in a language with a GC (golang?) or something crazy like rust? Or are C/C++ benefits just too high to ever give up?
And similarly, that simplicity is a benefit (ie, BoringSSL etc has some value).
It's hard to fault a project written in 2003 for not using Go, Rust, Haskell, etc... It is also hard to convince people to do a ground up rewrite of code that is seemingly working fine.
> It is also hard to convince people to do a ground up rewrite of code that is seemingly working fine.
I think this is an understatement, considering that it's a core cryptographic library. It appears to have gone through at least five audits (though none since 2010), and includes integration with hardware cryptographic accelerators.
Suggesting a tabula rasa rewrite of NSS would more likely be met with genuine concern for your mental well-being, than by incredulity or skepticism.
That’s just it though, it never was. That C/C++ code base is like a giant all-brick building on a fault line. It’s going to collapse eventually, and your users/the people inside will pay the price.
What's somewhat interesting is memory safety is not a totally new concept.
I wonder if memory safety had mattered more, whether other languages might have caught on a bit more, developed more etc. Rust is the new kid, but memory safety in a language is not a totally new concept.
The iphone has gone down the memory unsafe path including for high sensitivity services like messaging (2007+). They have enough $ to re-write some of that if they had cared to, but they haven't.
Weren't older language like Ada or Erlang memory safe way back?
I think a good approach could be what curl is doing. AFAIK they are replacing some security-critical parts of their core code with Rust codebases, importantly without changing the API.
1. Does the program need to be fast or complicated? If so, don't use a scripting language like Python, Bash, or Javascript.
2. Does the program handle untrusted input data? If so, don't use a memory-unsafe language like C or C++.
3. Does the program need to accomplish a task in a deterministic amount of time or with tight memory requirements? If so, don't use anything with a garbage collector, like Go or Java.
By 'complicated' in point 1, do you mean 'large'? Because a complex algorithm should be fine -- heck, it should be better in something like Python because it's relatively easy to write, so you have an easier time thinking about what you're doing, avoid making a mistake that would lead to an O(n³) runtime instead of the one you were going for, takes less development time, etc.
I assume you meant 'large' because, as software like Wordpress beautifully demonstrates, you can have the simplest program (from a user's perspective) in the fastest language but by using a billion function calls for the default page in a default installation, you can make anything slow. Using a slow language for large software, if that's what you meant to avoid then I agree.
And as another note, point number 2 basically excludes all meaningful software. Not that I necessarily disagree, but it's a bit on the heavy-handed side.
I suspect Ada would make the cut, with the number of times it's been referenced in these contexts, but I haven't actually taken the time to learn Ada properly. It seems like a language before its time.
Re 3, people have known how to build real-time GCs since like the 70s and 80s. Lots of Lisp systems were built to handle real-time embedded systems with a lot less memory that our equivalent-environment ones have today. Even Java was originally built for embedded. While it's curious that mainstream GC implementations don't tend to include real-time versions (and for harder guarantees need to have all their primitives documented with how long they'll execute for as a function of their input, which I don't think Rust has), it might be worth it to schedule 3-6 months of your project's planning to make such a GC for your language of choice if you need it. If you need to be hard real time though, as opposed to soft, you're likely in for a lot of work regardless of what you do. And you're not likely going to be building a mass-market application like a browser on top of various mass-market OSes like Windows, Mac, etc.
If your "deterministic amount of time" can tolerate single-digit microsecond pauses, then Go's GC is just fine. If you're building hard real time systems then you probably want to steer clear of GCs. Also, "developer velocity" is an important criteria for a lot of shops, and in my opinion that rules out Rust, C, C++, and every dynamically typed language I've ever used (of course, this is all relative, but in my experience, those languages are an order of magnitude "slower" than Go, et al with respect to velocity for a wide variety of reasons).
3b. Does your program need more than 100Mb of memory?
If no, then just use a GC'd language and preallocate everything and use object pooling. You won't have GC pauses because if you don't dynamically allocate memory, you don't need to GC anything. And don't laugh. Pretty much all realtime systems, especially the hardest of the hard real time systems, preallocate everything.
Answering a question with a sincere question: if the answer to 3 is yes to deterministic time, but no to tight memory constraints, does Swift become viable in question 4? I suspect it does, but I don’t know nearly enough about the space to say so with much certainty.
To provide some context for my answer, I’ve seen, first hand, plenty of insecure code written in python, JavaScript and ruby, and a metric ton - measured in low vulnerabilities/M LoC - of secure code written in C for code dating from the 80s to 2021.
I personally don’t like the mental burden of dealing with C any more and I did it for 20+ years, but the real problem with vulnerabilities in code once the low hanging fruit is gone is the developer quality, and that problem is not going away with language selection (and in some cases, the pool of developers attached to some languages averages much worse).
Would I ever use C again? No, of course not. I’d use Go or Rust for exactly the reason you give. But to be real about it, that’s solving just the bottom most layer.
C vulnerabilities do have a nasty habit of giving the attacker full code execution though, which doesn’t tend to be nearly so much of a problem in other languages (and would likely be even less so if they weren’t dependant on foundations written in C)
It’s not lost on me that the organization that produced NSS also invented Rust. That implies the knowledge of this need is there, but it’s not so straightforward to do.
> Or are C/C++ benefits just too high to ever give up?
FFI is inherently memory-unsafe. You get to rewrite security critical things from scratch, or accept some potentially memory-unsafe surface area for your security critical things for the benefit that the implementation behind it is sound.
This is true even for memory-safe languages like Rust.
The way around this is through process isolation and serializing/deserializing data manually instead of exchanging pointers across some FFI boundary. But this has non-negligible performance and maintenance costs.
Maybe this specific problem needs attention. I wonder, is there a way we can make FFI safer while minimizing overhead? It'd be nice if an OS or userspace program could somehow verify or guarantee the soundness of function calls without doing it every time.
If we moved to a model where everything was compiled AOT or JIT locally, couldn't that local system determine soundness from the code, provided we use things like Rust or languages with automatic memory management?
Writing a small wrapper that enforces whatever invariants are needed at the FFI boundary is much, much easier to do correctly than writing a whole program correctly.
You are never going to get 100% memory safety in any program written in any language, because ultimately you are depending on someone to have written your compiler correctly, but you can get much closer than we are now with C/C++.
C/C++ don’t really have “benefits”, they have inertia. In a hypothetical world where both came into being at the same time as modern languages no one would use them.
Sadly, I’m to the point that I think a lot of people are going to have to die off before C/C++ are fully replaced if ever. It’s just too ingrained in the current status quo, and we all have to suffer for it.
this is C code.
stuff like void*, union and raw arrays do not belond in modern C++.
while C++ is compatible with C it provides ways to write safer code that C doesn't.
writing C code in a C++ project is similar to writing inline assembly.
Are you suggesting that this crypto library would not be possible or practical to be built with rust? What features of C enable this library which Rust does not?
There is no time machine to bring rust back to when this was created, but as far as I know, there is no reason it shouldn't be Rust if it was made today.
How big is too big? I haven't run into any size issues writing very unoptimized Go targeting STM32F4 and RP2040 microcontrollers, but they do have a ton of flash. And for that, you use tinygo and not regular go, which is technically a slightly different language. (For some perspective, I wanted to make some aspect of the display better, and the strconv was the easiest way to do it. That is like 6k of flash! An unabashed luxury. But still totally fine, I have megabytes of flash. I also have the time zone database in there, for time.Format(time.RFC3339). Again, nobody does that shit on microcontrollers, except for me. And I'm loving it!)
Full disclosure, Python also runs fine on these microcontrollers, but I have pretty easily run out of RAM on every complicated Python project I've done targeting a microcontroller. It's nice to see if some sensor works or whatever, but for production, Go is a nice sweet spot.
I wondered this when I recently saw that flatpak is written in C. In particular for newer projects that don't have any insane performance constraints I wonder why people still stick to non memory-managed languages.
Dumb question: Do we need to use C++ anymore? Can we just leave it to die with video games? How many more years of this crap do we need before we stop using that language. Yes I know, C++ gurus are smart, but, you are GOING to mess up memory management. You are GOING to inject security issues with c/c++.
If you drop the parts of C++ that are that way because of C it is a much safer language. Weird and inconsistent, but someone who is writing C++ wouldn't make the error in the code in question any more than they would in rust. In C++ we never is unbounded arrays, just vectors and other bounded data structures.
I often see students asking a C++ question and when I tell then that is wrong they respond that their professor has banned vector. We have a real problem with bad teachers in C++, too many people learn to write C that builds with a C++ compiler and once in a while has a class.
C/C++ is great for AI/ML/Scientific computing because at the end of the day, you have tons of extremely optimized libraries for "doing X". But the thing is, in those use cases your data is "trusted" and not publicly accessible.
Similarly in trading, C/C++ abounds since you really do have such fine manual control. But again, you're talking about usage within internal networks rather than publicly accessible services.
For web applications, communications, etc.? I expect we'll see things slowly switch to something like Rust. The issue is getting the inertia to have Rust available to various embedded platforms, etc.
FWIW, Go absolutely would not stop you writing unbounded data into a bounded struct. Idiomatic Go would be to use byte slices which auto-resize, unlike idiomatic C, but you still have to do it.
Go would stop this from being exploitable. You might be able to make a slice larger than it is "supposed" to be, but it won't overwrite anything else because Go will be allocating new memory for your bigger slice.
But this is hardly a big claim for Go. The reality is that of all current languages in use, only C and C++ will let this mistake happen and have the consequence of overwriting whatever happens to be in the way. Everything else is too memory safe for this to happen.
Maybe this is not so much about switching individual existing projects to Rust, but about switching the "industry".
Some new projects are still written in C.
The same way you would write code written in 2021 over to Rust: by rewriting it from the ground up.
Auto-translation won't work because Rust won't allow you to build it in the same way you would have built it in C. It requires a full up redesign of the code to follow the Rust development model.
librsvg 1.0 was in 2001. Federico Mena-Quintero started switching it to Rust in 2017 (well technically October 2016), the rewrite of the core was finished early 2019, though the test suite was only finished converting (aside from the C API tests) late 2020.
But Mozilla is already using Rust. They are a major proponent of Rust, so if they did switch to a memory safe language it would seem like Rust would be the most likely choice for them.
In 1992 C++ was a decade old and still very niche.
In 2002 Python were both about a decade old and very niche.
In 2005 Javascript was a decade old and still very niche (only used on some webpages, the web was usable without javascript for the most part).
I think it's safe to say that all of them went on to enjoy quite a bit of success/widespread use.
Some languages take off really fast and go strong for a long time (php and java come to mind).
Some languages take off really fast and disappear just as fast (scala, clojure).
Some languages get big and have a long tail, fading into obscurity (tcl, perl).
Some languages go through cycles of ascendancy and descendancy (FP languages come to mind for that).
Dismissing a language because of it's adoption rate seems kinda silly - no one says "don't use python because it wasn't really popular til it had existed for over a decade".
Actually, we are talking the creators of rust here. The same guys who were owning it with the idea to rewrite the entire browser in it. The more plausible reason might be that the rewrite to rust haven't advanced to this component yet.
A decade seems like an appropriate amount of time for a language to mature and take off. Ruby was very niche for 10 years until rails came out. Rust now seems to be spreading pretty steadily and smaller companies are trying it out.
I guess the ecosystem that might make a language attractive was not built overnight. I'm not sure looking at the popularity since an initial release is the best way to measure how good a language is for a particular purpose.
There are two buffers and one size -- the amount of memory to copy.
There should be PORT_Memcpy2(pDest, destSize, pSource, numBytesToCopy) (or whatever you want to call it) which at least prompts the programmer to account for the size destination buffer.
Then flag all calls to PORT_Memcpy and at least make a dev look at it.
(Same for the various similar functions like strcpy, etc.)
Someone could do that, but the point of having the dest buffer size is to at least give the programmer a chance to try to get it right.
I also wonder if a linter could notice that the dest buffer size passed isn’t the actual size of the buffer. (That leads the the next problem in the code, if you look at the definition of that buffer, so that’s good.)
Readit News
Reasons why this buffer overflow wasn't caught earlier despite doing all the right things
And then to give those reasons:
- "each component is fuzzed independently" ... "This fuzzer might have produced a SECKEYPublicKey that could have reached the vulnerable code, but as the result was never used to verify a signature, the bug could never be discovered."
- "There is an arbitrary limit of 10000 bytes placed on fuzzed input. There is no such limit within NSS; many structures can exceed this size. This vulnerability demonstrates that errors happen at extremes"
- "combined [fuzzer] coverage metrics [...]. This data proved misleading, as the vulnerable code is fuzzed extensively but by fuzzers that could not possibly generate a relevant input."
The conclusion is, of course, to fix those problems if your code base also has them, but also "even extremely well-maintained C/C++ can have fatal, trivial mistakes".
No offense to the many bright and capable people who have worked hard on the C/C++ language, tools, compilers, libraries, kernels, etc over the years, but we will someday look back on it as asbestos and wonder why we kept at it for so damn long.
> No offense to the many bright and capable people who have worked hard on the C/C++ language, tools, compilers, libraries, kernels, etc over the years, but we will someday look back on it as asbestos and wonder why we kept at it for so damn long.
We won't wonder at all. We will understand that those people are the ONLY ones that stepped up to the task over 50 years to write this kind of software, organize standards bodies for their languages and platforms, get their software packaged as part of mainstream operating systems and out into the world, deal with patches from users, and help millions of other people make a living, enable the internet to happen, etc.
We will wonder why with all the millions of lines of C/C++ reference code available to be perused and then rewritten in Rust, Pascal, C#, Zig or Nim, and the vociferousness of their advocates, why that didn't happen in a reasonable timeframe.
We will wonder why all the Lisp and Haskell programmers who sneer down their nose at working C/C++ programmers in forums like this on a daily basis, didn't get off their asses with their One True Language (TM) and come to the worlds rescue.
The answer will be: these people aren't doers - they are talkers. It's one thing to get a toy proof of concept OS in your favourite language of choice that supports like 5 drivers. It's another thing to contribute to and build an ecosystem depended on by millions of developers daily. C/C++ people may not always be the sharpest tools in the shed, and they may be a dime a dozen. But they know how to organize themselves in loose groups of more than just a few developers, and work with other people.
We may have issues with the quality of what they ship on a frequent basis, but at least they ship instead of posting endless comments on internet forums about how it's all the others peoples fault.
Maybe. But there is good reason for C dominance - it's low level and close to systems as in "syscalls and stuff". And we need that level of systems control, not only for max performance but also for not loosing what is available in hw and os.
Asm is the other option to have full functionality :) Maybe it's just case of available libraries but still C is curently only one option for feature completness. And on low levels all you have is bytes and bits and calls and that allows to do everything with bits, bytes and calls - Core Wars style - and most of languages try to prevent or easy use of that by cutting possibilities, eg. "goto is so nono !11".
And yes, C is not perfect and bullshit even grows, eg. in last years NULL checks are "optimized-out"...
C actually should be improved and not fucked up into forced disuse...
That's why.
Such crazyness, like why?
Otherwise, the same story could be told as a generic software testing joke: "unit-tests are short-sighted and coverage lies", i.e. an "extremely well-maintained codebase, with extensive unittest, >98% test coverage and constantly scanned by all-static-analyzers-you-may-come-up" can have fatal, trivial bugs.
I've never understood that to mean that he wasn't in favor of automated testing, only that it's got its limits. In this case, they now know a test case that was missing.
Is that special? Are there buffer overflow bugs that are not security bugs? It could be just my bubble as a security consultant, since (to me) "buffer overflow" assumes remote code execution is a given. It's not my area of expertise, though, so perhaps indeed not all reachable buffer overflows are security issues. (Presuming no mitigations, of course, since those are separate from the bug itself.)
They enable refactoring code in ways not possible without them.
This is the one that seemed short sighted to me. It's a completely arbitrary (and small!) limit that blinded the fuzzer to this very modest sized buffer overflow.
Deleted Comment
Assuming that something is well-maintained because it is important is pure wishful thinking. Absent a specific detailed high quality process, or an audit that they conform to a well-established process that has demonstrably produced objectively high-quality output in a large percentage of audited implementations of that process (thus establishing nearly every instance of the audited process -> high quality output) all evidence indicates that you should assume that these code bases are poorly maintained until proven otherwise[3]. And, even the ones that are demonstrably maintained usually use very low quality processes as demonstrated by the fact that almost nobody working on those projects would be comfortable using their processes on safety-critical systems [4] which is the minimum bar for a high quality process (note the bar is "believe it is okay for safety-critical"). In fact, most would be terrified of the thought and comfort themselves knowing that their systems are not being used in safety-critical systems because they are absolutely not taking adequate precautions, which is a completely reasonable and moral choice of action as they are not designing for those requirements, so it is totally reasonably to use different standards on less important things.
[1] https://news.ycombinator.com/item?id=25919235
[2] https://github.com/sudo-project/sudo/graphs/contributors
[3] https://xkcd.com/2347/
[4] https://xkcd.com/2030/
[...] NSS was one of the very first projects included with oss-fuzz [...]
[...] Mozilla has a mature, world-class security team. They pioneered bug bounties, invest in memory safety, fuzzing and test coverage. [... all links to evidence ...]
Did Mozilla have good test coverage for the vulnerable areas? YES.
Did Mozilla/chrome/oss-fuzz have relevant inputs in their fuzz corpus? YES.
Is there a mutator capable of extending ASN1_ITEMs? YES.
I don't think at any point anyone assumed anything.
By contrast, NSS is maintained by Mozilla as part of Firefox, and, furthermore, its level of concern is deep into the "we don't want to enable certain cipher suites, and we have very exacting certificate validation policies that we are part of the effort in defining"--that is to say, NSS isn't a "somebody else's problem" dependency for Mozilla but a very "Mozilla's problem" dependency.
That said, this is CERT_VerifyCertificate, not mozilla::pkix, and since this is not used in Firefox's implementation of certificate validation, I would expect that this particular code in the library would be less well-maintained than other parts. But the whole library itself wouldn't be in the same camp as OpenSSL.
I wasn't under the impression anybody who knew the project ever really thought that. Some other people may have assumed that as a default if they hadn't looked into it.
This article spells out a whole bunch of reasoning why this particular library was well maintained though. There's a difference between reasoning based on evidence and assumptions.
With the benefit of a little hindsight, that does feel rather like saying "please try not to find any problems involving overflows".
OTOH file formats often include sizes of fields, which a fuzzer will set to arbitrarily high values. This tests (some) handling of overly large inputs without files being actually that large.
https://www.mozilla.org/en-US/security/advisories/mfsa2021-5... notes "This vulnerability does NOT impact Mozilla Firefox. However, email clients and PDF viewers that use NSS for signature verification, such as Thunderbird, LibreOffice, Evolution and Evince are believed to be impacted.".
*…with, I suppose, potential exceptions in performance-critical code when you control and trust the input; I don’t believe that this code qualifies on either count.
Because you can't.
It seems to me linters have been flagging this kind of thing since forever. This code is using a wrapper, "PORT_memcpy", so a default ruleset isn't going to flag it.
So here I guess no one noticed PORT_memcpy == memcpy (or maybe noticed but didn't take the initiative to add a lint rule or deprecation entry or just created an issue to at least port existing code).
So, just passing the size of the destination is something that you can still get wrong, in the case of data more complicated than just a single buffer.
[Edit: You can also design an API to be very misleading, even if it has a length parameter...]
For example, maybe your function is only called from another one that performs the appropriate bound checks, so checking again becomes redundant. After a simple refactoring, you can end up exposing your function, and screw up.
Languages like Rust, of course, provide basic memory safety out of the box, but most importantly they also provide means to package unsafe code under safe API and debug it once and for all. And ecosystem of easy to use packages help reusing good code instead of reinventing your own binary buffers every single damn time, as it's usually done in C.
So maybe it's not the unsafeness itself, but rather inability to build powerful reusable abstractions that plagues C? Everyone has to step on the same rake again and again and again.
We continue to be reminded that it's hard to write fully memory secure code in a language that is not memory secure?
And by hard, I mean, very hard even for folks with lots of money and time and care (which is rare).
My impression is that Apple's imessage and other stacks also have memory unsafe languages in the api/attack surface, and this has led to remote one click / no click type exploits.
Is there a point at which someone says, hey, if it's very security sensitive write it in a language with a GC (golang?) or something crazy like rust? Or are C/C++ benefits just too high to ever give up?
And similarly, that simplicity is a benefit (ie, BoringSSL etc has some value).
I think this is an understatement, considering that it's a core cryptographic library. It appears to have gone through at least five audits (though none since 2010), and includes integration with hardware cryptographic accelerators.
Suggesting a tabula rasa rewrite of NSS would more likely be met with genuine concern for your mental well-being, than by incredulity or skepticism.
That’s just it though, it never was. That C/C++ code base is like a giant all-brick building on a fault line. It’s going to collapse eventually, and your users/the people inside will pay the price.
I wonder if memory safety had mattered more, whether other languages might have caught on a bit more, developed more etc. Rust is the new kid, but memory safety in a language is not a totally new concept.
The iphone has gone down the memory unsafe path including for high sensitivity services like messaging (2007+). They have enough $ to re-write some of that if they had cared to, but they haven't.
Weren't older language like Ada or Erlang memory safe way back?
Or generate C from a safe GP language, eg C targeting Scheme such as Chicken Scheme / Bigloo / Gambit.
People have been shipping software in memory safe languages all this time, since way before stack smashing was popularized in Phrack, after all.
1. Does the program need to be fast or complicated? If so, don't use a scripting language like Python, Bash, or Javascript.
2. Does the program handle untrusted input data? If so, don't use a memory-unsafe language like C or C++.
3. Does the program need to accomplish a task in a deterministic amount of time or with tight memory requirements? If so, don't use anything with a garbage collector, like Go or Java.
4. Is there anything left besides Rust?
I assume you meant 'large' because, as software like Wordpress beautifully demonstrates, you can have the simplest program (from a user's perspective) in the fastest language but by using a billion function calls for the default page in a default installation, you can make anything slow. Using a slow language for large software, if that's what you meant to avoid then I agree.
And as another note, point number 2 basically excludes all meaningful software. Not that I necessarily disagree, but it's a bit on the heavy-handed side.
If no, then just use a GC'd language and preallocate everything and use object pooling. You won't have GC pauses because if you don't dynamically allocate memory, you don't need to GC anything. And don't laugh. Pretty much all realtime systems, especially the hardest of the hard real time systems, preallocate everything.
1. What are the people going to implement this an expert in?
Choose that. Nothing else matters.
Deleted Comment
I personally don’t like the mental burden of dealing with C any more and I did it for 20+ years, but the real problem with vulnerabilities in code once the low hanging fruit is gone is the developer quality, and that problem is not going away with language selection (and in some cases, the pool of developers attached to some languages averages much worse).
Would I ever use C again? No, of course not. I’d use Go or Rust for exactly the reason you give. But to be real about it, that’s solving just the bottom most layer.
There's nothing crazy about rust.
If your example was ATS we'd be talking.
FFI is inherently memory-unsafe. You get to rewrite security critical things from scratch, or accept some potentially memory-unsafe surface area for your security critical things for the benefit that the implementation behind it is sound.
This is true even for memory-safe languages like Rust.
The way around this is through process isolation and serializing/deserializing data manually instead of exchanging pointers across some FFI boundary. But this has non-negligible performance and maintenance costs.
Maybe this specific problem needs attention. I wonder, is there a way we can make FFI safer while minimizing overhead? It'd be nice if an OS or userspace program could somehow verify or guarantee the soundness of function calls without doing it every time.
If we moved to a model where everything was compiled AOT or JIT locally, couldn't that local system determine soundness from the code, provided we use things like Rust or languages with automatic memory management?
You are never going to get 100% memory safety in any program written in any language, because ultimately you are depending on someone to have written your compiler correctly, but you can get much closer than we are now with C/C++.
Sadly, I’m to the point that I think a lot of people are going to have to die off before C/C++ are fully replaced if ever. It’s just too ingrained in the current status quo, and we all have to suffer for it.
You don't need to explain to Mozilla about rewriting code from C/C++ to Rust.
Also, as far as I know, a full replacement for C doesn't exists yet.
There is no time machine to bring rust back to when this was created, but as far as I know, there is no reason it shouldn't be Rust if it was made today.
Full disclosure, Python also runs fine on these microcontrollers, but I have pretty easily run out of RAM on every complicated Python project I've done targeting a microcontroller. It's nice to see if some sensor works or whatever, but for production, Go is a nice sweet spot.
It is true that it’s not something you just get for free, you have to avoid certain techniques, etc. But rust can fit just fine.
It's more convenient than C. It's easier to use (at the cost of safety) compared to Rust.
Perhaps this will change if I know rust better. But for now C++ is where it's at for me for this niche.
I often see students asking a C++ question and when I tell then that is wrong they respond that their professor has banned vector. We have a real problem with bad teachers in C++, too many people learn to write C that builds with a C++ compiler and once in a while has a class.
Similarly in trading, C/C++ abounds since you really do have such fine manual control. But again, you're talking about usage within internal networks rather than publicly accessible services.
For web applications, communications, etc.? I expect we'll see things slowly switch to something like Rust. The issue is getting the inertia to have Rust available to various embedded platforms, etc.
https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=rust
I read "double free", "denial of service", "out-of bounds read", "NULL pointer dereference", etc...
And that's a list of vulnerabilities found for a language that is barely used compared to C/C++ (in the real world).
It won't change. C/C++ dominates and will dominate for a very long time.
Yep
But this is hardly a big claim for Go. The reality is that of all current languages in use, only C and C++ will let this mistake happen and have the consequence of overwriting whatever happens to be in the way. Everything else is too memory safe for this to happen.
I can see situations where I could probably get go to crash, but not sure how I get go to act badly.
Note: Not a go / Haskell / C# expert so understanding is light here.
Auto-translation won't work because Rust won't allow you to build it in the same way you would have built it in C. It requires a full up redesign of the code to follow the Rust development model.
So... carefully and slowly.
Deleted Comment
Deleted Comment
In 1982 C was a decade old and still very niche.
In 1992 C++ was a decade old and still very niche.
In 2002 Python were both about a decade old and very niche.
In 2005 Javascript was a decade old and still very niche (only used on some webpages, the web was usable without javascript for the most part).
I think it's safe to say that all of them went on to enjoy quite a bit of success/widespread use.
Some languages take off really fast and go strong for a long time (php and java come to mind).
Some languages take off really fast and disappear just as fast (scala, clojure).
Some languages get big and have a long tail, fading into obscurity (tcl, perl).
Some languages go through cycles of ascendancy and descendancy (FP languages come to mind for that).
Dismissing a language because of it's adoption rate seems kinda silly - no one says "don't use python because it wasn't really popular til it had existed for over a decade".
There are two buffers and one size -- the amount of memory to copy.
There should be PORT_Memcpy2(pDest, destSize, pSource, numBytesToCopy) (or whatever you want to call it) which at least prompts the programmer to account for the size destination buffer.
Then flag all calls to PORT_Memcpy and at least make a dev look at it. (Same for the various similar functions like strcpy, etc.)
I also wonder if a linter could notice that the dest buffer size passed isn’t the actual size of the buffer. (That leads the the next problem in the code, if you look at the definition of that buffer, so that’s good.)
Otherwise that's a potential read out of bounds.