If you have a reproducible test case that runs reasonably quickly, then I think printf debugging is usually just as good as a "real" debugger, and a lot easier. I typically have my test output log open in one editor frame. I make a change to the code in another frame, save it, my build system immediately re-runs the test, and the test log immediately refreshes. So when the test isn't working, I just keep adding printfs to get more info out of the log until I figure it out.
This sounds so dumb but it works out to be equivalent to some very powerful debugger features. You don't need a magical debugger that lets you modify code on-the-fly while continuing the same debug session... just change the code and re-run the test. You don't need a magical record-replay debugger that lets you go "back in time"... just add a printf earlier in the control flow and re-run the test. You don't need a magical debugger that can break when a property is modified... just temporarily modify the property to have a setter function and printf in the setter.
Most importantly, though, this sort of debugging is performed using the same language and user interface I use all day to write code in the first place, so I don't have to spend time trying to remember how to do stuff in a debugger... it's just code.
BUT... this is all contingent on having fast-running automated tests that can reproduce your bugs. But you should have that anyway.
> Most importantly, though, this sort of debugging is performed using the same language and user interface I use all day to write code in the first place, so I don't have to spend time trying to remember how to do stuff in a debugger... it's just code.
Absolutely! Running a commandline debugger adds additional context I have to keep in my head (syntax for all the commands, output format etc), that actively competes with context required to debug my code. Printfs work just fine without incurring that penalty, granted this argument applies less to IDE debuggers because their UX is usually intuitive.
> But for many bugs, getting to a reproduction is already more than half the battle.
Today I am working on a bug where a token expires after 2 hours and then we fail to request a new one instead we just keep on using the now expired one, which (of course) doesn’t work. I have a script to reproduce it but it takes 2 hours to run. It would be great if there was some configuration knob to turn down the expiry just for this test so we can reproduce it faster - but there isn’t because nobody thought of that.
Adding print statements, rebuilding and rerunning the program and figuring out where in the logs the bug showed up this time can a lot more tedious than setting a (possibly conditional) breakpoint in a reverse-execution debugger and doing "reverse-continue".
The crashes/bugs I deal with are rarely straight down failures, they are often the 1 out of 100 runs kind, so printf debugging is the only way to go really. And I used to be big on using debuggers, but now I’m horribly out of practice.
> The crashes/bugs I deal with are rarely straight down failures, they are often the 1 out of 100 runs kind, so printf debugging is the only way to go really.
Another thing I’ve found helpful, is to write out a “system state dump” (say in JSON) to a file whenever certain errors happen. Like we had a production system that was randomly hanging and running out of DB connections. So now whenever the DB connection pool is exhausted, it dumps a JSON file to S3 listing the status of every DB connection, including the stack dump of the thread that owns it, the HTTP request that thread is serving, the user account, etc. Once we did that, it went from “we don’t understand why this application is randomly falling over” to “oh this is the thing that is consistently triggering it”
When it writes a dump, it then starts a “lockout period” in which it won’t write any further dumps even if the error reoccurs. Don’t want to make a meltdown worse by getting bogged down endlessly writing out diagnostics.
Record-and-replay debuggers are often better than anything else for debugging intermittent failures: run the program with recording many times until you eventually get the bug, then debug that recording at your leisure; you'll never have to waste time reproducing it again.
Can you share an example? In my whole career, I have only seen one or two of them, but most of my work is CRUD type of stuff, not really gaming or systems programming where such a thing might happen.
I would say that like... saying "OK so `a.b` looks like this, what does `a.b.c` look like?" is a very nice flow with deep and messy objects (especially when working in languages with absolutely garbage default serialization logic like Javascript)
But even with a debugger, there's still loads of value of sitting up, moving from writing a bunch of single line statements all over, and to writing a real test harness ASAP to not have to rely on the debugger.
For any non-trivial problem, you'll often very quickly appreciate a properly formatted output stack, and that output will be shaped to the problem you are looking at. Very hard for an in-process debugger to have the answer for you there immediately.
Being serious about debugging can even get into things like writing actual new entrypoints (scripts with arguments and whatnot) and things like adding branches togglable through environment variables.
I think a lot of people's mindset in debugging is "if I walk the tightrope and put in _one more hack_ I'll find my answer" and it gets more and more precarious. Debugging is always a bit of an exercise of mental gymnastics, but if you practice being really good at printf debugging _and_ configuring program flow easily, you can be a lot less stressed out.
Like if you think your problem is going to take more than 20 minutes to debug, you probably should start writing a couple helper functions to get you on the right foot.
One of the hardest bugs I've investigated required the extreme version of debugging with printf: sprinkling the code with dump statements to produce about 500GiB of compressed binary trace, and writing a dedicated program to sift through it.
The main symptom was a non-deterministic crash in the middle of a 15-minute multi-threaded execution that should have been 100% deterministic. The debugger revealed that the contents of an array had been modified incorrectly, but stepping through the code prevented the crash, and it was not always the same array or the same position within that array. I suspected that the array writes were somehow dependent on a race, but placing a data breakpoint prevented the crash. So, I started dumping trace information. It was a rather silly game of adding more traces, running the 15-minute process 10 times to see if the overhead of producing the traces made the race disappear, and trying again.
The root cause was a "read, decompress and return a copy of data X from disk" method which was called with the 2023 assumption that a fresh copy would be returned every time, but was written with the 2018 optimization that if two threads asked for the same data "at the same time", the same copy could be returned to both to save on decompression time...
But I consider myself lucky that the issue could be reproduced on a local machine (arguably, one with 8 cores and 64GiB RAM) and not only on the 32 core, 256GiB RAM server. Having to work remotely on a server would have easily added another week of investigation.
I've gotten an OS to run on a new platform with a debugging tool portfolio consisting of a handful of LEDs and a pushbutton. After getting to the point where I could printf() to the console felt like more than anyone could ever ask for.
Anecdote aside, it certainly doesn't hurt to be able to debug things without a debugger if it comes to that.
Since most of my work is in distributed systems, I find the advice to never printf downright laughable.
"Oh sure, lemme just set a breakpoint on this network service. Hm... Looks like my error is 'request timed out', how strange."
That having been said: there are some very clever solutions in cloud-land for "printf" debugging. (Edit: forgot this changed names) Snapshot Debugger (https://github.com/GoogleCloudPlatform/snapshot-debugger) can set up a system where some percentage of your instances are run in a breakpointed mode, and for some percentage of requests passing through the service, they can log relevant state. You can change what you're tracking in realtime by adding listeners in the source code view. Very slick.
Whatever works so I can fix it and be home on time. I will even print (in paper) the code and step through it with a pen. Again, whatever works.
Also, will we ever move forward from these sort of discussions? Back when I was a mechanic no one argued about basics troubleshooting strategies. We just aimed to learn them and apply them all (as necessary).
> Whatever works so I can fix it and be home on time. I will even print (in paper) the code and step through it with a pen.
I’ve found before that sometimes I can’t see what’s wrong with the code on my screen but I can when I print it out. I think the printed page activates different regions of the brain compared to looking at a computer screen
You should try other things that force the brain to begin interpreting something again as new input; e.g., I have found success with changing the font, or the colour scheme, or looking at a diff rendition of a change, or looking at the file with less(1), or even reading the thing in reverse order, provoking a similarly new context where I see stuff I won't have seen staring at the editor for an hour previously. Another thing printing usually induces is that you simply stop looking at it for at least a few minutes before looking again, and a break often helps too!
Printing is obviously fine if you like doing that, but I've found there are lots of ways to shake yourself loose and more thoroughly review your own writing.
Based on the "lucky 10k" and the size of the internet, likely not. I'm sure if you find a mechanics enthusiast forum you will in fact find these relatively trivial arguments. It's just the nature of the beast.
I’d say programmers are taught to nitpick away at arguments by default. The nature of the code review is to verify that the code I wrote meets certain (opinions) standards. Hence nitpicking is built into the profession. Maybe one day it’ll improve. Till then, I’ll continue arguing about everything there is :-)
25 years ago I worked on a port of PC -> Playstation 1 game. We did not had proper devkits, but the yaroze model ("amateur", allowing for "printf"-debugging of sorts)
Long story short, our game worked as long as the printfs we had were kept, we had macro to remove them (in "Release/Ship") but the game crashed.
The crash was due to side-effect of printf clearing some math errors.... So here you go!
agree entirely that both techniques are useful and should be learned by programmers
one thing I think the "just do print debugging" folks miss is what a good teaching tool a visual debugger is: you can learn about what a call stack really is, step through conditionals, iterations, closures, etc and get a feel for how they really work
at some level being a good programmer means you can emulate the code in your head, and a good visual developer can really help new programmers develop that skill, especially if they aren't naturals
i emphasize the debugger in all the classes i teach for this reason
> at some level being a good programmer means you can emulate the code in your head, and a good visual developer can really help new programmers develop that skill [emphasis added]
I think you meant "a good visual debugger". And I agree completely.
A visual debugger isn't just a tool for fixing bugs.
It's also a tool for understanding the code.
At my last job, the codebase was so complicated that you could spend hours scratching your head over what was going on in a function, especially how the code got to that function and what data the calling functions had that led to this point.
Of course you could add print or log statements, but then the question is what to print! And which calling functions needed more print statements.
With a visual debugger, I could just set a breakpoint in the confusing function, see all the data it had, and also move up the stack to see what data all the calling functions had.
There are cases where you need print debugging. I added one feature that worked perfectly locally and on a test server, but failed in a Jenkins job (ironically running the job on that same test server).
That was a case where I added print statements throughout the code, just to see how far it got when running locally vs. under Jenkins.
There are many ways to debug a problem. It is wise to be familiar with all of them and know what to use when.
I find that a lot of this discussion just melts away if you are aware of the options that are available and then take your pick. Why, sometimes I've switched between debuggers and printfs as many as six times before breakfast.
If you don't know what a debugger does though that's something you should really get on ASAP. Likewise if you can't figure out how to get log messages out of your thing. Really all there is to it, figure out what you want to do after than and spend your time actually doing something productive instead of getting in a stupid holy war on the internet about it.
If using the debugger is less efficient than using printf then it's a symptom of a wider problem. To be clear though, "printf debugging" is not the same thing as adding structured debug logs to your service and enabling it on demand via some config. Most production services should never be logging unstructured output to stdout. Printf debugging is just throwing out some text to stdout, and it generally means running unit tests locally and iterating through an (add printf, run test, add more printf) loop until the bug is found. Unfortunately it's the default way to debug locally reproducible bugs for so many engineers. So while I don't see the point in not using printf for purely ideological reasons, I avoid building software where this is the simplest option and use the service's structured logger if not easily reproducible. I also generally think it's bad to default to printf debugging in the way I have defined it here and find that competent use of debugger is more often a faster way to debug.
Readit News
This sounds so dumb but it works out to be equivalent to some very powerful debugger features. You don't need a magical debugger that lets you modify code on-the-fly while continuing the same debug session... just change the code and re-run the test. You don't need a magical record-replay debugger that lets you go "back in time"... just add a printf earlier in the control flow and re-run the test. You don't need a magical debugger that can break when a property is modified... just temporarily modify the property to have a setter function and printf in the setter.
Most importantly, though, this sort of debugging is performed using the same language and user interface I use all day to write code in the first place, so I don't have to spend time trying to remember how to do stuff in a debugger... it's just code.
BUT... this is all contingent on having fast-running automated tests that can reproduce your bugs. But you should have that anyway.
Absolutely! Running a commandline debugger adds additional context I have to keep in my head (syntax for all the commands, output format etc), that actively competes with context required to debug my code. Printfs work just fine without incurring that penalty, granted this argument applies less to IDE debuggers because their UX is usually intuitive.
Ideally, yes. But for many bugs, getting to a reproduction is already more than half the battle. And a debugger can help you with that.
Today I am working on a bug where a token expires after 2 hours and then we fail to request a new one instead we just keep on using the now expired one, which (of course) doesn’t work. I have a script to reproduce it but it takes 2 hours to run. It would be great if there was some configuration knob to turn down the expiry just for this test so we can reproduce it faster - but there isn’t because nobody thought of that.
Another thing I’ve found helpful, is to write out a “system state dump” (say in JSON) to a file whenever certain errors happen. Like we had a production system that was randomly hanging and running out of DB connections. So now whenever the DB connection pool is exhausted, it dumps a JSON file to S3 listing the status of every DB connection, including the stack dump of the thread that owns it, the HTTP request that thread is serving, the user account, etc. Once we did that, it went from “we don’t understand why this application is randomly falling over” to “oh this is the thing that is consistently triggering it”
When it writes a dump, it then starts a “lockout period” in which it won’t write any further dumps even if the error reoccurs. Don’t want to make a meltdown worse by getting bogged down endlessly writing out diagnostics.
But even with a debugger, there's still loads of value of sitting up, moving from writing a bunch of single line statements all over, and to writing a real test harness ASAP to not have to rely on the debugger.
For any non-trivial problem, you'll often very quickly appreciate a properly formatted output stack, and that output will be shaped to the problem you are looking at. Very hard for an in-process debugger to have the answer for you there immediately.
Being serious about debugging can even get into things like writing actual new entrypoints (scripts with arguments and whatnot) and things like adding branches togglable through environment variables.
I think a lot of people's mindset in debugging is "if I walk the tightrope and put in _one more hack_ I'll find my answer" and it gets more and more precarious. Debugging is always a bit of an exercise of mental gymnastics, but if you practice being really good at printf debugging _and_ configuring program flow easily, you can be a lot less stressed out.
Like if you think your problem is going to take more than 20 minutes to debug, you probably should start writing a couple helper functions to get you on the right foot.
Deleted Comment
The main symptom was a non-deterministic crash in the middle of a 15-minute multi-threaded execution that should have been 100% deterministic. The debugger revealed that the contents of an array had been modified incorrectly, but stepping through the code prevented the crash, and it was not always the same array or the same position within that array. I suspected that the array writes were somehow dependent on a race, but placing a data breakpoint prevented the crash. So, I started dumping trace information. It was a rather silly game of adding more traces, running the 15-minute process 10 times to see if the overhead of producing the traces made the race disappear, and trying again.
The root cause was a "read, decompress and return a copy of data X from disk" method which was called with the 2023 assumption that a fresh copy would be returned every time, but was written with the 2018 optimization that if two threads asked for the same data "at the same time", the same copy could be returned to both to save on decompression time...
But I consider myself lucky that the issue could be reproduced on a local machine (arguably, one with 8 cores and 64GiB RAM) and not only on the 32 core, 256GiB RAM server. Having to work remotely on a server would have easily added another week of investigation.
Deleted Comment
Anecdote aside, it certainly doesn't hurt to be able to debug things without a debugger if it comes to that.
"Oh sure, lemme just set a breakpoint on this network service. Hm... Looks like my error is 'request timed out', how strange."
That having been said: there are some very clever solutions in cloud-land for "printf" debugging. (Edit: forgot this changed names) Snapshot Debugger (https://github.com/GoogleCloudPlatform/snapshot-debugger) can set up a system where some percentage of your instances are run in a breakpointed mode, and for some percentage of requests passing through the service, they can log relevant state. You can change what you're tracking in realtime by adding listeners in the source code view. Very slick.
Time travel debugging (https://en.wikipedia.org/wiki/Time_travel_debugging) can help with this because it separates "recording" (i.e. reproducing the bug) from "replaying" (i.e. debugging).
Breakpoints only need to be set in the replay phase, once you've captured a recording of the bug.
Also, will we ever move forward from these sort of discussions? Back when I was a mechanic no one argued about basics troubleshooting strategies. We just aimed to learn them and apply them all (as necessary).
You're correct, the real issue comes down to
1. making sure those printf statements don't wind up in prod, spilling potentially sensitive data or corrupting a data stream
2. making sure that non-printf tooling is built so that only printf debugging isn't used
We tend to get caught up in false dichotomies.
https://news.ycombinator.com/item?id=42579472
I’ve found before that sometimes I can’t see what’s wrong with the code on my screen but I can when I print it out. I think the printed page activates different regions of the brain compared to looking at a computer screen
Printing is obviously fine if you like doing that, but I've found there are lots of ways to shake yourself loose and more thoroughly review your own writing.
Present them as options. "If you like X, you may also like Y." Leave it to the audience to discriminate when to apply the tool.
Long story short, our game worked as long as the printfs we had were kept, we had macro to remove them (in "Release/Ship") but the game crashed.
The crash was due to side-effect of printf clearing some math errors.... So here you go!
one thing I think the "just do print debugging" folks miss is what a good teaching tool a visual debugger is: you can learn about what a call stack really is, step through conditionals, iterations, closures, etc and get a feel for how they really work
at some level being a good programmer means you can emulate the code in your head, and a good visual developer can really help new programmers develop that skill, especially if they aren't naturals
i emphasize the debugger in all the classes i teach for this reason
I think you meant "a good visual debugger". And I agree completely.
A visual debugger isn't just a tool for fixing bugs.
It's also a tool for understanding the code.
At my last job, the codebase was so complicated that you could spend hours scratching your head over what was going on in a function, especially how the code got to that function and what data the calling functions had that led to this point.
Of course you could add print or log statements, but then the question is what to print! And which calling functions needed more print statements.
With a visual debugger, I could just set a breakpoint in the confusing function, see all the data it had, and also move up the stack to see what data all the calling functions had.
There are cases where you need print debugging. I added one feature that worked perfectly locally and on a test server, but failed in a Jenkins job (ironically running the job on that same test server).
That was a case where I added print statements throughout the code, just to see how far it got when running locally vs. under Jenkins.
There are many ways to debug a problem. It is wise to be familiar with all of them and know what to use when.
If you don't know what a debugger does though that's something you should really get on ASAP. Likewise if you can't figure out how to get log messages out of your thing. Really all there is to it, figure out what you want to do after than and spend your time actually doing something productive instead of getting in a stupid holy war on the internet about it.