I noticed that the two bugs they linked [1] [2] are both due to the distros introducing bugs by applying patches -- one for hurd support (!) -- that were not shipped by the upstream projects that the distros were repackaging. As we have discussed earlier [3] I think the way distros inject themselves into this software development process produces these bad outcomes due to getting the incentives wrong.
Arch Linux does not do this, instead preferring to upstream patches [0]:
> Arch Linux defines simplicity as without unnecessary additions or modifications. It ships software as released by the original developers (upstream) with minimal distribution-specific (downstream) changes: patches not accepted by upstream are avoided, and Arch's downstream patches consist almost entirely of backported bug fixes that are obsoleted by the project's next release.
It's one of the core things that has kept me on Arch as a daily driver, even long after I've lost the urge to endlessly tweak my system configuration. I can trust that the software I use is simply vanilla upstream software with little or no modifications, and that's a great advantage when it comes to filing upstream bug reports and working on patches. In addition, it means the Arch Wiki is fairly general in its applicability, and effort spent documenting software for Arch can apply equally well to, for example, Void Linux (which also has this "vanilla software" philosophy).
As a former Debian enthusiast (I even helped staff a Debian booth at a conference once!) who also tends to be conservative with new technology and who is consequently also skeptical of all these random Linux distros, I eventually tried out Arch and found it ... super awesome, I really love it!
I highly recommend it to anyone else like me, who is generally cranky about new things. They did a really great job with Arch. This policy you mention is a great example of what I like about it.
>For example, at some point, Debian updated the fontconfig package by backporting an upstream fix for a memory leak. Unfortunately, the fix contained a bug that would crash Firefox and possibly other software too. We spotted the new crash only six days after the change landed in Debian sources and only a couple of weeks afterwards the issue had been fixed both upstream and in Debian. We sent reports and fixes to other projects too including Mesa, GTK, glib, PCSC, SQLite and more.
That sounds a lot like Linus' "many eyes make all bugs shallow" idea working as intended.
Even more to the point is that these bugs were found in Debian's unstable/testing distribution. Reiterating that the release process is happening as it should and the bugs are being found by people who volunteered to help test the software.
Yes, that's one of the benefits. Firefox has a very large user-base compared to other FOSS projects so we will often spot bugs that others haven't noticed simply thanks to the sheer volume of crash reports we get.
This third bug was not due to Debian patches. And in fact the two Debian patch cases in the article were included to make the point that they can now catch issues caused by distribution patches, not to claim that most of the issues they find are caused in this way. Assuming on the basis of an article written to make a wholly different point that because 2/3 were the result of distribution patches that this must be a huge problem in general for Linux software development just shows your bias on this issue.
I like and use Arch Linux on my desktops. But I'd defend the choice by Debian to patch. Most of Debian's changes are intended to improve support for certain setups or introduce bug or security fixes that upstream hasn't backported. These are both good things. Furthermore, problems that are created tend to be caught while they're still in unstable or sid: if I'm not mistaken that's exactly what happened in these cases, which is the process working as intended. (Mozilla is certainly free to disregard bug reports from Debian users if they feel that Debian's patching process is simply causing too many problems.)
I think it's a good thing that there's enough information to easily correlate things like crash reports from the latest bleeding-edge packages with uploads of dependency packages:
> the libfontconfig1 package was updated on Debian on the 21st of April and the first crash we have on record was sent on the 22nd. Here's the changelog:
I think that's a good argument for this kind of crash-telemetry, working in conjunction with Linux distributions. And it helped, in this case, that the dependencies could be updated independently from Firefox.
On the other hand, the libdrm hurd patch breaking things on Linux seems like an excellent example of how portability to obscure platforms has a non-zero cost, and it isn't as simple as "just accept portability patches".
This is exactly why historically Mozilla has required distributors to distribute an unmodified version of Mozilla software to use Mozilla trademarks such as the name Firefox, which (unlike the code) are not freely available.
Unfortunately the specific bugs here are in library packages that Firefox uses and those have no such restrictions.
The blog post could be titled "Why the Linux distribution model is totally broken for end user software".
As a linux user I appreciate the work, but it's tough to read this blog post and not think about the wasted effort that could be used to fix the bugs in the upstream software itself.
Really? Because I just think about how impossible this would be on Windows, where shared or external libs are inscrutable, as are OS build chains, version differences, etc etc.
Are you saying switching to static binaries would help because... they are harder to patch? If I'm not mistaken, Debian and most Linux distros build Firefox from source. In this instance, if Firefox was only available as a statically-compiled binary, it would still be vulnerable to the issue Debian was patching!
> the way distros inject themselves into this software development process produces these bad outcomes
As a former maintainer of packages in an enterprise Linux ecosystem, I agree. But...the patches are often meant to solve a dependency mismatch and the maintainers can't easily pull related dependencies forward without breaking other applications. It's great that other distros like Arch just simply upstream the patches - it's the right thing to do after all - but that takes time that the enterprise distributions don't always have.
Agree wholeheartedly. As far as I am concerned the whole repo/package manager model is the root cause of most of the reasons the Linux Desktop experience is as awful as it is.
It's actually pretty great. Linux distributions have actual human maintainers that everybody trusts and they generally do the right thing. The fact random developers don't have direct access to people's installs is a major feature. If you want to release a Linux package, you should have to work with distribution maintainers in order to actually integrate your package with their ecosystem.
It's not like PyPI/rubygems/npm/whatever where any random person can make an account and start pushing packages. The maintainers have to actually trust you.
No, the reason is SW development process - developers which introduce big changes between library versions ( why do i need fontconfig 4.3.2_pl6 for example). When the library is a moving target any package which depend on it and also the security of the system become a moving target. Unfortunately Firefox does the same ( they discontinued ESR because it is boring fixing bugs).
Firefox is great on Linux. It kinda whoops Chrome's ass for what it provides (though Chrome on Linux is a nice experience), and Mozilla has consistently been one of the most charitable organizations towards the Linux foundation. Here's to another decade of Firefox and an open web!
If you exclude most hardware acceleration on majority of Linux PCs that currently run FF and if you exclude following standards that good and normal software does[1].
Both issues exist nearly decades now. "Great"? Absolutely not.
Chrome's sandbox is far superior to Firefox's from a security standpoint. I think the best bet is to run Ungoogled Chromium on Linux; it's got even less telemetry than Firefox and is more secure.
You're focusing on technicalities, but the real issue is political/social in my opinion. If Firefox did not exist, what other powers exist to prevent Google from shaping the Web for its own benefit more than it already has?
I personally don't even care whether or how much feature parity Firefox has or whether its GPU rendering is 20% slower. Not supporting Firefox comes at our own loss.
It's been a long time since I have seen a firefox crash on Linux.
What's killing me is the memory leak that just renders the whole computer unusable and almost frozen. Almost because if I can grab a terminal and killall firefox then I get the machine back.
I've been encountering a similar memory leak on Windows, so I don't know if it's really related to Linux.
What I do know is that the way Linux GUI distros deal with low memory situations is absolutely garbage. The new systemd OOM daemon that's shipping with systemd 248 will hopefully improve this situation, but for now I'm left running nohang[1] on my dev machine, where I'm consistently running out of RAM (IntelliJ + tons of Java frameworks + huge React code base + a web browser all take up way too much space!). Enabling zRAM also seems to work, but I haven't measured the effectiveness of that yet. On my home PC with double the ram (32GiB) everything runs smoothly, but the way Linux on the desktop deals with OOM situations is still atrocious. Besides, none of these tools should be consuming such ungodly amounts of RAM anyway.
For server situations, Linux handles OOMs better than Windows, IMO. On desktop, the same strategy just doesn't work and only makes working on it more painful.
About year ago, i started to run Firefox in memory-limited cgroup (to 4 GB), so when it eats all the memory, it just locked itself (by thrashing pages at full SSD speed) and not the whole machine. It is done by:
Same here... I very rarely see Firefox crashing or a tab crashing on Linux. It happens once in a very rare while that I see a tab crashing but the whole browser never crashes.
Memory leaks can be bad indeed and here are two tricks that work fine for me: either run Firefox inside a Docker container... Docker container on which I put CPU and RAM quota. It's IMO way easier than try to put resources quota directly on Firefox. The other one: Firefox has zero issues reopening all my tabs, so I simply kill it from the terminal when I see that it's going wild on memory. Now I've got 16 GB and Firefox rarely eats it all and it's usually not sudden: it's a slow bleed over the course of a few days... So when I get to something like 10 GB of memory used by just one Firefox instances (I typically run several Firefox instances, from different user accounts) , I just kill it and restart it, while choosing to reopening all the tabs (usually tens of tabs).
I'm sure there are other methods too but this works fine in my case so I haven't looked much into it.
I also notice the mem leaks, but I can usually leave my Firefox process running until there is the next update to install.
What works for me is having the `about:memory` tab open and clicking on "Minimize memory usage" at the end of the day when I suspend the machine. And, I have a lot of tabs and windows open, strewn over various virtual desktops. Though most tabs are not loaded but more expensive bookmarks, but I have no add-on which actively unloads tabs.
When things get bad, check about:memory and check out which process is using all the memory, and for what. It may just be some Web application you use that has a leak of its own.
To be fair to Mozilla this firefox bug having the effect of rendering the "whole computer unusable and almost frozen" is 100% a Linux bug/feature. Programs have bugs and memory leaks. The OS has to handle cases where they do. This is what the OS does when it happens here.
> prevent it from asking for a restart after update
This isn't just nagging. The API between the main browser context and content processes isn't stable between builds. This means that Firefox can't spin up a new content process if the update has replaced the executable with a different version. Therefore Firefox has the tough choice of trying anyways and risking misbehaviour (including possibly security bugs) or forcing the user to restart the browser process so that it can spawn new content processes.
It also depends on how your distribution updates Firefox. For most distributions they have this problem, however NixOS doesn't because the new version is installed in a new directory.
IIUC Firefox's self-updater also doesn't have this issue, and I think it supports Linux.
> This means that Firefox can't spin up a new content process if the update has replaced the executable with a different version.
Couldn't it spin up another one from the old version by doing execve("/proc/self/exe", ...)? Or by keeping a template process around with all the libraries loaded from which a child process can then be forked.
I would absolutely love a shortcut manager for Firefox. For example I use this shortcut enough that something like Ctrl+Shift+Q would be useful. (And the dozens of other things that I would like to rebind, including custom shortcuts).
I haven't been able to disable it, but the option "Warn you when quitting the browser" at the top of preferences has already saved me numerous times. Not as perfect as disabling, but far better than nothing.
I've never really had an issue with restarting Firefox, it remembers all my tabs when I restart it either after an update or just a close/open. Perhaps it's a setting I toggled years ago and have since forgotten but I'm not sure what the complaint is about restarting after an update?
At least for me after FF updates itself it won't let you open a new tab or anything until it restarts. According to this Reddit post this is due to it being updated by the package manger so maybe it's a Linux only issue?: https://old.reddit.com/r/firefox/comments/jwx54y/firefox_for...
It remembers the tabs, but it loses my vertical position within the tabs. When reading articles or a thread on some forum, that's quite annoying. Even Pocket doesn't save my progress on a desktop but does on my phone.
It's nothing that would make me stop using it, just an annoyance that I wish I didn't have to deal with on a semi-regular basis.
Per a sibling comment, browser.quitShortcut.disabled in about:config works. For me, I just never disabled the option to ask before quitting when more than one tab is open, so that dialog always catches me before I quit by accident.
Sadly it's not enough. If I have one tab open, I still don't want ctrl-q to kill Firefox; and it doesn't count pinned tabs as open tabs, so it will still kill even if you have several pinned tabs open. It's a really, really dumb shortcut to have on by default and I'm thrilled it's finally dead.
>"When it comes to Linux things work differently than on other platforms: most of our users do not install our builds, they install the Firefox version that comes packaged for their favourite distribution.
This posed a significant problem when dealing with stability issues on Linux: for the majority of our crash reports, we couldn’t produce high-quality stack traces because we didn’t have the required symbol information. The Firefox builds that submitted the reports weren’t done by us. To make matters worse, Firefox depends on a number of third-party packages (such as GTK, Mesa, FFmpeg, SQLite, etc.). We wouldn’t get good stack traces if a crash occurred in one of these packages instead of Firefox itself because we didn’t have symbols for them either.
To address this issue, we started scraping debug information for Firefox builds and their dependencies from the package repositories of multiple distributions: Arch, Debian, Fedora, OpenSUSE and Ubuntu. Since every distribution does things a little bit differently, we had to write distro-specific scripts that would go through the list of packages in their repositories and find the associated debug information
"
This is why I'm glad that a lot of Linux software is moving towards distro agnostic containerized applications maintained by developers. Linux already does not have a large market share and if you have to deal with a dozen distro-specific quirks it seems hardly economical.
> This is why I'm glad that a lot of Linux software is moving towards distro agnostic containerized applications
To me, it entirely depends on the specific packaging job, not even who is doing it. Entirely too often, they ignore distro convention, and can introduce bugs and other security issues through ignorance, laziness or mistake.
Long term, flatpack and its ilk will devalue distro differences and put pressure on uniformity. Whether or not you consider this is a good thing may depend on your opinion of/relationship to RedHat/IBM.
For me, if it isn't in the Debian repos, I'm probably going to ignore it.
> Entirely too often, they ignore distro convention
Can you clarify what you mean by this?
I have used both the flatpak and rpm (on Fedora) version of Firefox and don't really notice a difference. Perhaps Firefox is just one of the packages that does distro convention properly?
I'm not holding my breath. The Linux Standard Base is 20 years old now. That was supposed to get us there. If Canonical and Red Hat cared, they'd probably go through the Linux Foundation instead of creating competitors to AppImage. Speaking of, AppImage is from 2004. So... see you guys in 20 more years.
Unfortunately they are solutions designed for a platform and ecosystem with a lot of suck built-in. For my money, AppImage has the most sane approach, but without support from the wider community they are still a bit of a pain.
One of the issues of running an open-source C++ project is the difficulty in obtaining stack traces from users - hoping to re-use the work that Mozilla did here for Firefox, so thanks team for paving the path there for others.
You're very welcome! If you're interested in the topic we've got a working group [1] and a very active channel on chat.mozilla.org [2]. Besides the actual stability work we've also been busy either rewriting some of this tooling in Rust (see [3] and [4]) as well as contributing to Sentry's excellent symbolic crate which we leverage during symbol extraction.
How do you extract symbols from official binary builds of Arch? According to official documentation (https://wiki.archlinux.org/title/Debug_-_Getting_Traces and bug report threads I've seen), Arch lacks both debug packages and symbols bundled with binary packages. As a result, if you want to debug an app crash on Arch, you need to quit the program, recompile the app from source with your makepkg.conf set to !strip, then try to reproduce the bug in your new build.
Only public symbols and unwinding information is extracted from Arch packages because they don't have debuginfo like other distros. It's better than nothing.
Well that's unfortunate to hear :( Hoping Arch can fix releasing debug symbols for official binary builds soon... but I don't expect it to happen soon.
Also how does unwinding info help during debugging? Does gdb use it? Do I need special reverse-engineering tools to extract useful information?
Readit News
[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1679430
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=1633467
[3] https://news.ycombinator.com/item?id=26216917
> Arch Linux defines simplicity as without unnecessary additions or modifications. It ships software as released by the original developers (upstream) with minimal distribution-specific (downstream) changes: patches not accepted by upstream are avoided, and Arch's downstream patches consist almost entirely of backported bug fixes that are obsoleted by the project's next release.
It's one of the core things that has kept me on Arch as a daily driver, even long after I've lost the urge to endlessly tweak my system configuration. I can trust that the software I use is simply vanilla upstream software with little or no modifications, and that's a great advantage when it comes to filing upstream bug reports and working on patches. In addition, it means the Arch Wiki is fairly general in its applicability, and effort spent documenting software for Arch can apply equally well to, for example, Void Linux (which also has this "vanilla software" philosophy).
[0]: https://wiki.archlinux.org/title/Arch_Linux
I highly recommend it to anyone else like me, who is generally cranky about new things. They did a really great job with Arch. This policy you mention is a great example of what I like about it.
My desktop has looked and behaved the same for almost a decade.
I migrate to a new machine by dd'ing from backups and haven't installed fresh in years.
These things are what keep me around.
>For example, at some point, Debian updated the fontconfig package by backporting an upstream fix for a memory leak. Unfortunately, the fix contained a bug that would crash Firefox and possibly other software too. We spotted the new crash only six days after the change landed in Debian sources and only a couple of weeks afterwards the issue had been fixed both upstream and in Debian. We sent reports and fixes to other projects too including Mesa, GTK, glib, PCSC, SQLite and more.
That sounds a lot like Linus' "many eyes make all bugs shallow" idea working as intended.
This is misleading and in fact not even strictly true. They link three bugs, not two: https://bugzilla.mozilla.org/show_bug.cgi?id=1633459
This third bug was not due to Debian patches. And in fact the two Debian patch cases in the article were included to make the point that they can now catch issues caused by distribution patches, not to claim that most of the issues they find are caused in this way. Assuming on the basis of an article written to make a wholly different point that because 2/3 were the result of distribution patches that this must be a huge problem in general for Linux software development just shows your bias on this issue.
I like and use Arch Linux on my desktops. But I'd defend the choice by Debian to patch. Most of Debian's changes are intended to improve support for certain setups or introduce bug or security fixes that upstream hasn't backported. These are both good things. Furthermore, problems that are created tend to be caught while they're still in unstable or sid: if I'm not mistaken that's exactly what happened in these cases, which is the process working as intended. (Mozilla is certainly free to disregard bug reports from Debian users if they feel that Debian's patching process is simply causing too many problems.)
> the libfontconfig1 package was updated on Debian on the 21st of April and the first crash we have on record was sent on the 22nd. Here's the changelog:
I think that's a good argument for this kind of crash-telemetry, working in conjunction with Linux distributions. And it helped, in this case, that the dependencies could be updated independently from Firefox.
On the other hand, the libdrm hurd patch breaking things on Linux seems like an excellent example of how portability to obscure platforms has a non-zero cost, and it isn't as simple as "just accept portability patches".
Unfortunately the specific bugs here are in library packages that Firefox uses and those have no such restrictions.
As a linux user I appreciate the work, but it's tough to read this blog post and not think about the wasted effort that could be used to fix the bugs in the upstream software itself.
Earlier discussion: https://news.ycombinator.com/item?id=27009044
As a former maintainer of packages in an enterprise Linux ecosystem, I agree. But...the patches are often meant to solve a dependency mismatch and the maintainers can't easily pull related dependencies forward without breaking other applications. It's great that other distros like Arch just simply upstream the patches - it's the right thing to do after all - but that takes time that the enterprise distributions don't always have.
It's not like PyPI/rubygems/npm/whatever where any random person can make an account and start pushing packages. The maintainers have to actually trust you.
If you exclude most hardware acceleration on majority of Linux PCs that currently run FF and if you exclude following standards that good and normal software does[1].
Both issues exist nearly decades now. "Great"? Absolutely not.
[1]: https://bugzilla.mozilla.org/show_bug.cgi?id=259356
I personally don't even care whether or how much feature parity Firefox has or whether its GPU rendering is 20% slower. Not supporting Firefox comes at our own loss.
What's killing me is the memory leak that just renders the whole computer unusable and almost frozen. Almost because if I can grab a terminal and killall firefox then I get the machine back.
https://bugzilla.redhat.com/show_bug.cgi?id=1597028 something like that, not sure about the root cause. But I don't let imgur tab with running vid opened for too long now.
What I do know is that the way Linux GUI distros deal with low memory situations is absolutely garbage. The new systemd OOM daemon that's shipping with systemd 248 will hopefully improve this situation, but for now I'm left running nohang[1] on my dev machine, where I'm consistently running out of RAM (IntelliJ + tons of Java frameworks + huge React code base + a web browser all take up way too much space!). Enabling zRAM also seems to work, but I haven't measured the effectiveness of that yet. On my home PC with double the ram (32GiB) everything runs smoothly, but the way Linux on the desktop deals with OOM situations is still atrocious. Besides, none of these tools should be consuming such ungodly amounts of RAM anyway.
For server situations, Linux handles OOMs better than Windows, IMO. On desktop, the same strategy just doesn't work and only makes working on it more painful.
[1]: https://github.com/hakavlad/nohang
1. It comes preconfigured: https://build.opensuse.org/package/view_file/openSUSE:Factor...
2. Packaging adoption: https://repology.org/project/earlyoom https://repology.org/project/nohang
3. Written in C and unburdened with tangential features like desktop notifications, the daemon's process takes very little amount of memory.
In /etc/rc.local:
And then start it by script with:Memory leaks can be bad indeed and here are two tricks that work fine for me: either run Firefox inside a Docker container... Docker container on which I put CPU and RAM quota. It's IMO way easier than try to put resources quota directly on Firefox. The other one: Firefox has zero issues reopening all my tabs, so I simply kill it from the terminal when I see that it's going wild on memory. Now I've got 16 GB and Firefox rarely eats it all and it's usually not sudden: it's a slow bleed over the course of a few days... So when I get to something like 10 GB of memory used by just one Firefox instances (I typically run several Firefox instances, from different user accounts) , I just kill it and restart it, while choosing to reopening all the tabs (usually tens of tabs).
I'm sure there are other methods too but this works fine in my case so I haven't looked much into it.
What works for me is having the `about:memory` tab open and clicking on "Minimize memory usage" at the end of the day when I suspend the machine. And, I have a lot of tabs and windows open, strewn over various virtual desktops. Though most tabs are not loaded but more expensive bookmarks, but I have no add-on which actively unloads tabs.
[1] https://github.com/rfjakob/earlyoom
Now if I could only disable Ctrl+Q and prevent it from asking for a restart after update...
This isn't just nagging. The API between the main browser context and content processes isn't stable between builds. This means that Firefox can't spin up a new content process if the update has replaced the executable with a different version. Therefore Firefox has the tough choice of trying anyways and risking misbehaviour (including possibly security bugs) or forcing the user to restart the browser process so that it can spawn new content processes.
It also depends on how your distribution updates Firefox. For most distributions they have this problem, however NixOS doesn't because the new version is installed in a new directory.
IIUC Firefox's self-updater also doesn't have this issue, and I think it supports Linux.
Couldn't it spin up another one from the old version by doing execve("/proc/self/exe", ...)? Or by keeping a template process around with all the libraries loaded from which a child process can then be forked.
Good news! =)
Set browser.quitShortcut.disabled in about:config
That's awesome, just did this. But apparently you need to restart Firefox in order to apply the change. (Learned it the hard way)
But otherwise, FF has been really stable for years now
It's nothing that would make me stop using it, just an annoyance that I wish I didn't have to deal with on a semi-regular basis.
But I agree with you, I can't remember when was the last time I had a crash before these!
This posed a significant problem when dealing with stability issues on Linux: for the majority of our crash reports, we couldn’t produce high-quality stack traces because we didn’t have the required symbol information. The Firefox builds that submitted the reports weren’t done by us. To make matters worse, Firefox depends on a number of third-party packages (such as GTK, Mesa, FFmpeg, SQLite, etc.). We wouldn’t get good stack traces if a crash occurred in one of these packages instead of Firefox itself because we didn’t have symbols for them either.
To address this issue, we started scraping debug information for Firefox builds and their dependencies from the package repositories of multiple distributions: Arch, Debian, Fedora, OpenSUSE and Ubuntu. Since every distribution does things a little bit differently, we had to write distro-specific scripts that would go through the list of packages in their repositories and find the associated debug information
"This is why I'm glad that a lot of Linux software is moving towards distro agnostic containerized applications maintained by developers. Linux already does not have a large market share and if you have to deal with a dozen distro-specific quirks it seems hardly economical.
To me, it entirely depends on the specific packaging job, not even who is doing it. Entirely too often, they ignore distro convention, and can introduce bugs and other security issues through ignorance, laziness or mistake.
Long term, flatpack and its ilk will devalue distro differences and put pressure on uniformity. Whether or not you consider this is a good thing may depend on your opinion of/relationship to RedHat/IBM.
For me, if it isn't in the Debian repos, I'm probably going to ignore it.
Can you clarify what you mean by this?
I have used both the flatpak and rpm (on Fedora) version of Firefox and don't really notice a difference. Perhaps Firefox is just one of the packages that does distro convention properly?
Deleted Comment
Firefox on Android is a different matter and is unusable on some devices.
[1] https://wiki.mozilla.org/Data/WorkingGroups/CrashReporting [2] https://chat.mozilla.org/#/room/#crashreporting:mozilla.org [3] https://github.com/mozilla/dump_syms/ [4] https://github.com/luser/rust-minidump
It seems Firefox has a script at https://github.com/gabrielesvelto/symbol-scrapers/blob/maste..., but I haven't investigated what it does. In any case, this information should be integrated into the Arch Wiki.
Also how does unwinding info help during debugging? Does gdb use it? Do I need special reverse-engineering tools to extract useful information?