Geoff’s not wrong in invoking Bret Victor’s Learnable Programming argument that being able to track state over time is critical to debugging, and Geoff’s right that print debugging makes this easier than almost any existing step-debugger.
Bret’s deeper point, though, is that a major challenge in debugging is hidden state in general, and that variable state changing over time is just one example.
Not only is there a ton of hidden state in the execution of a program—the full execution trace PLUS state of every variable at each point in that trace—but there is also a ton of interpretation of that state that the programmer needs to do while debugging: “what does this sequence of events imply?” - “why is this pointer pointing here?” - etc.
Doing that interpretation is much easier when the programmer gets to selectively view that (again, HUGE amount of) hidden state. Print debugging gives the programmer complete control over what state is shown. No other debugger does that: they all show a ton of data and context (often useful!) and make certain operations easy (inspecting single variables! viewing call stack snapshot!), and these are often just the right things.
But sometimes they’re not. And often, when you start debugging, you don’t know if the fancy debuggers will be too much or not enough.
Print debugging gives you the power to write code to selectively view the (again, HUGE!) hidden state of your program, and this scales from the smallest code-tracing bug to the largest distributed systems.
Step debuggers, on the other hand, are essentially “no-code” debuggers — extremely useful for the purpose for which they are designed, still useful for adjacent purposes, and a great place to start if you know the tool well, but ultimately not as powerful if your needs exceed their capacities.
A good programmer will know how to use all these tools for what they’re best at!
Same thing with watch windows, memory views and the like. There are classes of problems that do well with printf but calling them "no-code" is vastly underselling them.
A data breakpoint is a great example of something useful that print doesn’t do well.
I also don't think they're nearly as no-code as you call out. VS' watch window has very few limitations compared to printf back when I was working on win32 things.
Also important to consider iteration time. I once worked on a system where adding a printf was a 20 minute process due to the need to heavily optimize for a memory constrained platform(scripting fit in 400kb block with the asset bake step).
Exactly!
Debuggers are very useful tools, and typically not as general-purpose as print. I don’t view “not as powerful” as a meaningful distinction, because it requires that you ask “powerful at what?” ---
VS’ watch window is great but (I assume) doesn’t work across distributed systems, etc. — as a general technique, print is universal in the sense that there are very few problems that can’t be diagnosed by modifying your code and printing some (possibly a manipulation!) of the hidden state. This is going to be harder than using a special-purpose tool designed for exactly your problem.
In the same way, “no-code” tools are typically better and/or easier than writing code to solve the same problem, but special-purpose.
In my domain which doesn't usually cover distributed systems printf can be worse because it introduces synchronization primitives that have caused race conditions to disappear(and that race condition causes second order heap corruption or the like). On one platform system memory was so small(8mb total) that each output to stdout went over the serial link slowing performance down to 1/20th of a realtime process under any real logging.
Like I said, different tools for different uses, and really depends on the context. If there was one size fits all then we'd just use that but the diversity of debugging tools I think shows that you need a variety of techniques to approach the problems we encounter.
We totally agree, and I'm not sure what we're arguing about--perhaps you can fill me in.
I'm arguing that print is almost always worse than any specialized tool. (After all, who would use a specialized tool worse than print?) There is not a one-size-fits-all tool, and print is not a one-size-fits-all tool.
Indeed almost every seasoned developer has a story about print failing. Whether it's the mysterious "Heisenbug" that disappears when you measure it (like the sync issues you mention) -- my personal story is when I was trying to debug a (class project) kernel scheduler. Printing to the console was so slow that by the time I'd printed anything, the scheduler had moved on to the next time slice!
It's worth nothing that "print debugging" is not literally just using the "print" function; it's a style of debugging that involves logging specific information using some logging function (usually, but not always, print) and then analyzing it after the fact (usually, but not always, by reading the printed output).
This strategy of "get data out, then analyze it" is the general form of print debugging, and in the small-memory case, or the sync Heisenbug case, this often means collecting data in the appropriate variables before outputting it to be visible. Isn't this still print debugging, even though it doesn't use a "print" function?
With print debugging your inserting the whole build + deploy + repro setup loop into your debugging, if that's a long time(say 20 minutes in one job I had with production hardware) you're in for a world of pain. I find that just about any other tool usually is an order of magnitude more efficient.
Also even the "step debugger" tools do the same thing you'd do with a print. LLVM for instance uses the IR JIT API to generate watch/eval values: https://releases.llvm.org/9.0.0/docs/ORCv2.html#use-cases
IMO you should relentlessly optimize your iteration times, that's the inner loop of development speed and print debugging fares pretty poorly in that area for all the reasons above.
Ah, that's fair.
> At least for me print debugging is a measure of last resort
Right, and I think this depends on the domain. For lots of mature environments, this makes sense -- there's been years for tooling to catch up to the kinds of bugs people run into, possibly corporate money being put into developing debugging tools, etc.
> IMO you should relentlessly optimize your iteration times, that's the inner loop of development speed and[...]
Agreed, though the effect on print debugging on iteration time is very environment-dependent.
> [...]print debugging fares pretty poorly in that area for all the reasons above
Adding console.log to a web app can be a trivial change (though of course reproducing app state is another issue) -- again very environment-dependent.
I'm not making the argument that people shouldn't use debuggers -- obviously if there's a good one that does what you need, it'd be silly not to use it. And good debuggers are great.
But what happens when you're working on a distributed system? Or multiple processes on a single system? Or...etc. etc. -- debuggers are almost always built to support a particular set of use cases for a particular set of domains. Step outside, and they're not usually as useful.
print always works. It's almost never the best. It's more flexible because it can almost always be used.
Option1: Print all of them, requires rebuild, would log 200000 lines every second. Unless i wrap the print inside conditions, requiring yet another rebuild.
Option2: Conditional breakpoint: user[i].department.balance <= expected_value. Bang, i can now inspect both the complete nested user-object, and the previous/next item in the list, other local and global variables, the call stack of how it reached there, the state of all other threads in this moment, and so on. With the really good debuggers like .NET or JVM I'm even able to rewind the execution pointer to the start of the function or hot-swap the code as it's running.
In short, a debugger allows you to see all state of the program at once, and you can retroactivelly choose what is relevant. As opposed to printf where you must select upfront. Maybe even more importantly it's lacking stack traces, unless you add a log and request-id at every function-enter/exit.
There is also the case of third party code which you can't edit to add printf to. Many environments usually by default give you pretty good context of the symbol names or even complete source code which you can step through (except c++ land where you'd need a debug-build of the lib, which isn't impossible either).
If you have some completely undocumented object oriented code with tons of relationships, peeling the state apart in a debugger is a good way to get an idea of what you even have that might be worth printing.
Lots of things but an obvious example is private member info.
Debugging via print allows you to step outside and peer in ie it is active.
Print debugging can be prone to bugs within itself which may cause additional ignorance about the potential bugs being diagnosed. How meta can you get? 8) There's also the effect of the effort of actually looking - that may or may not have an effect.
Anyway, the discussion here is largely ignorant of language and function. At the moment I spend time fiddling up Python and OpenSCAD scripts if I dig out a programming language. For me, print is really handy. For a Linux low level latency sensitive driver in highly optimised ASM n C I suspect this matter is moot.
Nope, print debugging does not always work. Please stop saying that. Sure, it's often useful and often works. But go try to debug a race condition, or bug in some locking mechanism with print statements. I'll wait as you add a bunch of print statements, and then learn the harsh lesson that you are now debugging a completely different program with very different performance characteristics that now doesn't deadlock where it used to deadlock because you slowed down one of the threads massively as it spends time it used to not spend logging stuff.
"Step debuggers, on the other hand, are essentially “no-code” debuggers — extremely useful for the purpose for which they are designed, still useful for adjacent purposes, and a great place to start if you know the tool well, but ultimately not as powerful if your needs exceed their capacities."
I guess you'd think this if you thought print debugging "always works". But I promise you it is equally (if not easier) to exceed the capabilities of print statements in some scenarios.
I’ve been trying to articulate this to myself for a long time. Thank you.
Yeah, I run through valgrind; 9 times out of 10 I don't even get to the point of starting the debugger or recompiling with prints. The remaining 1 time where valgrind output is clean, I go with printing. About half the time the prints aren't enough and then I fire up the debugger.
Also, it depends. I'm working on a legacy C# project where the previous dev read somewhere that passwords shouldn't be stored in plain-text, so when changing the stored DB credentials you have to set a breakpoint where the hash is calculated, change the variable holding the cleartext password to the new password you want to use, step the line that calculates the new hash, copy the text out of the watch window and finally paste that text into the file holding the credentials.
I do not know how this helps. I also don't care enough to write a c/line utility that generates the hash in a base-64 string, becasue we've changed the DB credentials for the webapp only once since he left. We may have changed it more often if we had an easy way to do so :-/
That seems to be the idea behind Glamorous Toolkit https://gtoolkit.com/
I'd be curious what it would take to add any of those concepts to existing debuggers for other languages.
Personally, I think folks should master the debugger _first_ and during all steps learning a programming language.
But similar to test-driven-development it’s a different way of thinking, and most books scarcely discuss the debuggers.
That being said, I do use print-debugging a lot too—in C++ a lot of functionality can be compiled-out, allowing one to, for instance, print hex dumps of serialized data going to the network.
On that note, there is a distinction between trace debugging that is part of the source code and general print statements that are hacked in and removed.
Also, pycharm isn't really what I would call a proper debugger yet, attaching remote running processes for example just doesn't work reliably yet and is very new anyways. Debugging embedded targets just doesn't work. Multithreading is iffy (but that's unfortunately normal in Python).
Having said that, it is absolutely a requirement when working on a project for any length of time (especially professionally) to set up and figure out a debugging environment, because it is significantly more productive than printing. But the startup cost is certainly there.
Being able to debug third party code in remote / hostile environments (even when its mixed with proprietary vendor code) is one of the things I like about Java.
In the Java case, for stanalone projects (i.e. not something deployed on a server) an if it is your own project and you don't do anything unreasonable it is mostly just set a breakpoint and hit "run with debugging".
Probably the least painful debugging experience I know.
Doing it for Tomcat/Tomee was slightly more advanced IMO but still utterly trivial compared to wrangling css or js ;-)
There are reasons why we "old folks" like Java so much despite its verboseness.
On the other hand... adding print statements can also invalidate certain optimizations (an excellent source of heisenbugs), so I'll never stop using debuggers either
Setting up remote debugging I’ll agree is more difficult than a local application, but each remote machine can automatically run startup commands and not require user input; commands can be run at particular places too (to print output etc) with conditional trace points, all while not impacting the code itself.
Main point is that folks don’t spend enough time learning the debugger, as print statements are easier. But using the debugger is a better practice in my opinion in the case where print statements are added just for a quick test, then removed.
Adding a 'breakpoint()' at the start of the program does get me into the debugger. I'll remember that for future (but, it's not easy to find by googling if you don't already know what you are looking for!)
For large unwieldy data-structures, you can go ipython: `import IPython; IPython.embed()` launches the ipython REPL from the calling line's context.
I use the latter a fair bit when spelunking around in other people's code. `pdb.set_trace()` lets you continue execution more easily.
We haven't had one hit production yet, but it came close. Print statement is a lot more harmless.
You download Intellij IDEA, run it, choose File->Open and select the build.gradle file, right click the main class and there's a Debug option.
I agree.
A similar statement is that using a debugger often does not work.
the sort of ratholes I've run into are: - debug build - proper symbols - interrupts and debugger - kernel and debugger - unfamiliarity with debugger - limitations of debugger
Without a proper debug build, you can't run the debugger effectively. You have to set up a whole debug environment.
Sometimes you need to do broad work to get proper symbols and stack trace information. By broad, I mean a debug build for everything.
I've also found many debug builds, apart from altering some behavior, they also turn on a lot of printfs.
If you're using interrupts/timers or debugging a kernel module, many times debuggers don't work or may alter, move or supress the problem.
and then there's the... I haven't used a debugger in 6 months, how do I do (very simple thing). And sometimes the debugger is just not the right tool or a tedious tool to use. "If I just load this one macro and somehow get the right address maybe I can decode this one kernel data structure..."
Personally, many times an ephereral printf isn't crude and meaningless, it's a precise scalpel getting to the core of the problem.
FWIW my experience debugging Python in VS Code required no setup and I've encountered zero issues.
``` import pdb; pdb.set_trace() ```
that's literally all you have to do at any point in your code. Run your code in the foreground of a terminal, and boom you have a debugger exactly where you want it.
You can edit the state and keep running after altering the program.
But usually I just end up printing a few things from the repl and figuring out what is fouled up.
These systems likely already have a way to get logs, but good luck getting a debugger to work there.
Yes I am dieing on this hill.
Debuggers aren't bad. But neither is printing. Knowing when to reach for them is probably a bit more key.
The problem is that the two print statements will only catch the bug if they are the right two, based on a correct hypothesis of what the bug is. Which, with a debugger, won’t require stepping, but setting two breakpoints, doing a run-to-breakpoint, and inspecting values.
Stepping is required when you are exploring behavior because you don’t have an easily testable hypothesis about the source of the bug.
As you note, debugger breakpoints aren't magically better than print statements when I'm investigating a hypothesis – I'm no more likely to put them the right place than I would have put print statements.
And then there's a class of problems that neither debugger nor print statements will help: many years ago a very junior co-worker was wondering why his C code was giving the wrong answer for some math. It took me pointing out that one of the numeric types he was using in his code was different from the rest (I think it was #defined elsewhere, in some library, as an integer type). When the compiler did the math it had to do some type coercing.
A debugger and watches on the values of concern absolutely will help with that (so will properly placed print statements), so its a really bad example. (Of course, strong typing helps even more with that particular case.)
Can't I also bisect with breakpoints?
The worst developers I've ever known always have their "this is the best way to do everything" hill they die on.
Same applies to other debuggers.
It is not only the debuggers, but also OS and language runtime tracing facilities like DTrace, eBPF, ETW, JFR, ....
Many devs aren't 10x because of QI, rather because they learn and make use of the tools available to increase knowledge about the platform.
So we get generations that use vim and Emacs like notepad, create Makefiles by copy-paste and barely know gdb beyond set breakpoint, run, step and continue.
Using C as example, but feel free to extrapolate to another language.
And no I am not exaggerating, this was the kind of students I would get into my lab when I spent a year as TA in 1999/2000, and naturally would have to get them up to speed into good programming practices.
It is an exaggeration. In practice, it is useful to apply both. Novices can get some insight using debugging, more experienced with code base people should exercise their understanding of the code and use well picked prints.
Also, I think what you suggest to do here is way harder to learn than printing.
I'd throw your comment back -- maybe all languages should be "debug first", make it as easy to get code into a debugger as it is to just build and run it.
Try debugging something in the embedded world, and you’ll see why a lot of bare metal programmers use printfs. Turns out timing is critical most of the time, so using a debugger hides a LOT of bugs from your eyes.
Debuggers are very useful, but so are prints, there just different tools, they have different purposes.
And then you have to figure out the syntax... does it want package names, class names, or (hello Intellij plugins!) need a fucking # at the beginning to be recognized.
And then you have stuff like a "helpful" IDE that by default only shows WARN and above levels without telling you somewhere "there might be stuff you don't see" like Chrome does.
For actual debuggers, shit is worse, across the board. Running in Docker is always a recipe for issues, not to mention many applications actively messing around with stuff like ports.
A system.out.println always ends up somewhere sane.
In C++ I often find the debugger doesn't find symbols on projects built with configure/Make. If I have a Java Gradle project I have no idea how to get it into a debugger. Python debuggers always seem fragile. Rust requires I install and use a "rust-gdb" script -- except on my current machine that doesn't work and I don't know why.
I'm sure in each case I'm sure I could get a debugger working given enough time, but the only error I've ever had in print debugging is failing to flush output before a crash, and it's never been hard to search for "how to flush output" in whatever language I'm current using.