I didn't really get into debuggers until (1) I was firmly in Windows, where you expect the GUI to work and the LI to be busted, and (2) I'd been burned too many times by adding debugging printfs() that got checked into version control and caused trouble.
Since then I've had some adventures with CLI debuggers, such as using gdb to debug another gdb, using both jdb and gdb on the same process at the same time to debug a Java/C++ system, automating gdb, etc. But there is the thing, as you say, is that there is usually some investment required to get the debugger working for a particular system.
With a good IDE I think Junit + debugging gives an experience in Java similar to using the REPL in a language like Python in that you can write some code that is experimental and experiment it, but in this case the code doesn't just scroll out of the terminal but ultimately gets checked in as a unit test.
> As personal choice, we tend not to use debuggers beyond getting a stack trace or the value of a variable or two. One reason is that it is easy to get lost in details of complicated data structures and control flow; we find stepping through a program less productive than thinking harder and adding output statements and self-checking code at critical places. Clicking over statements takes longer than scanning the output of judiciously-placed displays. It takes less time to decide where to put print statements than to single-step to the critical section of code, even assuming we know where that is. More important, debugging statements stay with the program; debugging sessions are transient.
I tend to agree with them on this. For almost all of the work that I do, this hypothesis-logs-exec loop gets me to the answer substantially faster. I'm not "trying to run the code forwards in my head". I already have a working model for the way that the code runs, I know what output I expect to see if the program is behaving according to that model, and I can usually quickly intuit what is actually happening based on the incorrect output from the prints.
[0] The unreasonable effectiveness of print debugging (349 points, 354 comments) April 2021 https://news.ycombinator.com/item?id=26925570
Hard to get wrong, tedious to type and a huge speed increase to visually scan the output.
Most of us aren't Brian Kernighan or Rob Pike.
I am very happy for people who are, but I am firmly at a grug level.
I insist that my students learn a visual debugger in my classes for this reason: what the "stack" really is, how a loop really executes, etc.
It doesn't replace thinking & print debugging, but it complements them both when done properly.
It's useful for a beginner e.g in a for loop to see how `i` changes at the end of the loop. And similarly with return values of functions and so on.
Perhaps been underappreciating the gap in mental intuition of the runtime between then and now and how much the debugger helped to bridge.
He said that him and Brian K would pair while debugging. As Rob Pike told it, he would often drive the computer, putting in print statements, rerunning the program and so on. Brian Kernighan would stand behind him and quietly just think about the bug and the output the program was generating. Apparently Brian K would often just - after being silent for awhile - say "oh, I think the bug is in this function, on this line" and sure enough, there it was. Apparently it happened so often enough that he thought Brian might have figured out more bugs than Rob did, even without his hands touching the keyboard.
Personally I love a good debugger. But I still think about that from time to time. There's a good chance I should step away from the computer more often and just contemplate it.
My guess is that:
- Debuggers are most useful when you have a very poor understanding of the problem domain. Maybe you just joined a new company or are exploring an area of the code for the first time. In that case you can pick up a lot of information quickly with a debugger.
- Print debugging is most useful when you understand the code quite well, and are pretty sure you've got an idea of where the problem lies. In that case, a few judicious print statements can quickly illuminate things and get you back to what you were doing.
Most people who work on crud services do not see any benefit from it, as there is practically nothing going on. Observing input, outputs and databases is usually enough, and when it's not a well placed log will suffice. Debuggers will also not help you in distributed environments, which are quite common with microservices.
The only real issue in debugging a distributed/multithreaded environment is that frequently there is a timeout somewhere that is going to kill one of the threads that you may have wanted to continue stepping through after debugging a different thread.
1. When you get a stack trace from a failure, without knowing anything else find the right level and make sure it has sensible error handling and reporting.
1a. If the bug is reproduced but the program experiences no failure & associated stack trace, change the logic such that if this bug occurs then there is a failure and a stack trace.
1b. If the failure is already appropriately handled but too high up or relevant details are missing, fix that by adding handling at a lower level, etc.
That is the first step, you make the program fail politely to the user and allow through some debug option recover/report enough state to explain what happened (likely with a combination of logging, stack trace, possibly app-specific state).
Often it can also be the last step, because you can now dogfood that very error handling to solve this issue along with any other future issue that may bubble up to that level.
If it is not enough you may have to resort to debugging anyway, but the goal is to make changes that long-term make the use of either debuggers or print statements unnecessary in the first place, ideally even before the actual fix.
In my case (a digital audio workstation), a debugger can be handy for figuring out stuff in the GUI code, but the "backend" code is essentially a single calltree that is executed up to a thousands times a second. The debugger just isn't much use there; debug print statements tend to be more valuable, especially if a problem would require two breakpoints to understand. For audio stuff, the first breakpoint will often break your connection to the hardware because of the time delay.
Being able to print stacktraces from inside the code is also immensely valuable, and when I am debugging, I use this a lot.
But the debugger is essential for all the hard stuff. I’ll take heap snapshots and live inside lldb for days tracking down memory alignment issues. And print statements can be either counterproductive at best, or completely nonexistent in embedded or GPU bound compute.
- John Carmack loves debuggers
- Debuggers are most useful when you have a very poor understanding of the problem domain
John Carmack knows his domain very well. He knows what he expects to see. The debugger gives him insight into what “other” developers are doing without having to modify their code.
For Carmack, managing the code of others the debug environment is their safe space. For Kernighan et al in the role of progenitorous developer it is the code itself that is the safe space.
When I'm unfamiliar with a codebase, or unfamiliar with a particular corner of the code, I find myself reaching for console debugging. Its a bit of a scattershot approach, I don't know what I'm looking for so I console log variables in the vicinity.
Once I know a codebase I want to debug line by line, walking through to see where the execution deviates from what I expected. I very frequently lean on conditional breakpoints - I know I can skip breaks until a certain condition is met, at which point I need to see exactly what goes wrong.
In scripting languages print debugging makes sense especially when debugging a distributed system.
Also logging works better than breaking when debugging multithreading issues imo.
I use both methods.
If I wanna use a debugger though, now that means a full recompile to build the project without optimizations, which probably takes many minutes. And then I'll have to hope that I can reproduce the issue without optimizations.
10 minutes.
>If I wanna use a debugger though, now that means a full recompile to build the project without optimizations, which probably takes many minutes.
Typically always compile with debug support. You can debug an optimized build as well. Full recompile takes up to 45 minutes.
The largest reason to use a debugger is the time to recompile. Kinda, I actually like rr a lot and would prefer that to print debugging.
Have you tried out the Mold linker? It might speed it up significantly.
> You can debug an optimized build as well.
Eh, not really. Working with a binary where all variables are optimized out and all operators are inlined is hell.
I don't know the exact amounts of time per phase, but you might change a header file and that will of course hurt you a lot more than 1 translation unit.
> Eh, not really. Working with a binary where all variables are optimized out and all operators are inlined is hell.
Yeah, but sometimes that's life. Reading the assembly and what not to figure things out.
>That must absolutely suck to work with.
Well, you know, I also get to work on something approximately as exciting as Chromium.
Yeah, which is why I was talking about source files. I was surprised that changing 1 source file (meaning re-compiling that one source file and then re-linking the project) takes 10 minutes. If you're changing header files then yeah it's gonna take longer.
I've worked on projects where incremental linking after touching the wrong .cpp file will take several minutes... and this is after I've optimized link times from e.g. switching from BFD to Gold, whereas before it might take 30 minutes.
A full build (from e.g. touching the wrong header) is measured in hours.
And that's for a single configuration x platform combination, and I'm often the sort to work on the multi-platform abstractions, meaning I have it even worse than that.
> If I wanna use a debugger though, now that means a full recompile to build the project without optimizations
You can use debuggers on optimized builds, and in fact I use debuggers on optimized builds more frequently than I do unoptimized builds. Granted, to make sense of some of the undefined behavior heisenbugs you'll have to understand disassembly, and dig into the lower level details when the high level ones are unavailable or confused by optimization, but those are learnable skills. It's also possible to turn off optimizations on a per-module basis with pragmas, although then you're back into incremental build territory.
Sometimes, you just really do need hundreds of thousands of lines of C++ (or equivalent) ...
I use the debugger in Intellij, using breakpoints which only log some context and do not stop the current thread. I can add / remove them without recompiling. There's no interruption in my flow (thus no excuse to check HN because "compiling...")
When I show this to colleagues they think it's really cool. Then they go back to using print statements anyway /shrug
Debugging kernel module issues on AL2 on bare metal ec2 with kprint. Issue does not reproduce in qemu. It happens
This is the "it's one banana Michael, how much could it cost, ten dollars?" of tech. I don't think I've ever worked on a nontrivial C++ project that compiled in three seconds. I've worked in plenty of embedded environments where simply the download-and-reboot cycle took over a minute. Those are the places where an interactive debugger is most useful .. and also sometimes most difficult.
(at my 2000s era job a full build took over an hour, so we had a machine room of ccache+distcc to bring it down to a single digit number of minutes. Then if you needed to do a full place and route and timing analysis run that took anything up to twelve hours)
No C++ project compiles in 3 seconds, but your "change a single source file and compile+link" time is often on the order of a couple of seconds. As an example, I'm working on a project right now where a clean build takes roughly 30 seconds (thanks to recent efforts to improve header include hygiene and move stuff from headers into source files using PIMPL; it was over twice that before). However, when I changed a single source file and ran 'time ninja -C build' just now, the time to compile that one file and re-link the project took just 1.5 seconds.
I know that there are some projects which are much slower to link, I've had to deal with Chromium now and then and that takes minutes just to link. But most projects I've worked with aren't that bad.
Concurrency bugs just suck to debug sometimes.
First that visual debugging was still small and niche, probably not suited to the environment at Bell Labs at the time, given they were working with simpler hardware that might not provide an acceptable graphical environment (which can be seen as a lot of the UNIX system is oriented around the manipulation of lines of text). This is different from the workplace where most game developers, including J. Carmack were, with access to powerful graphical workstations and development tools.
Secondly there’s also a difference on the kind of work achieved: the work on UNIX systems mostly was about writing tools than big systems, favoring composition of these utilities. And indeed, I often find people working on batch tools not using visual debuggers since the integration of tools pretty much is a problem of data structure visualization (the flow being pretty linear), which is still cumbersome to do in graphical debuggers. The trend often is inverted when working on interactive systems where the main problem actually is understanding the control flow than visualizing data structures: I see a lot more of debuggers used.
Also to keep in mind that a lot of engineers today work on Linux boxes, which has yet to have acceptable graphical debuggers compared to what is offered in Visual Studio or XCode.
In any case, my suggestion is to understand both approaches and boldly use them in the right circumstance. If the need arises, be a rebel and break out the debugger or be a rebel and add some printfs - just don't weakly follow some tribal ritual.
By keep just one party to the loan, and most important, by me offering collateral to the loan in the event I do not deliver, then it keeps enforcement more honest and possible.
Furthermore, the loan contract should be written in such a way that the game is judged by the ONE TIME sales performance of the game (no microtransactions) and not qualitative milestones like features or reviews. Lastly, I would add a piece of the contract that says two years after the game is released, it becomes fully open source, similar to the terms of the BSL.
This is the fairest thing to the players, the bank, and the developer, and it lets the focus be absolutely rock solid on shipping something fun ASAP.
1) We only support Linux. 2) We only support Vulkan (or OpenGL, but I prefer Vulkan) 3) Singleplayer only. 4) Engine features: RGBA Diffuse, specular, shadow, normal/bump, emission maps for materials only. A simple triangle rasterizer with forward rendering. Support for audio, dae animations/rigs/geometry. Physics collisions through an octree and capsule and basic linear algebra. Basic first person/third person cameras. 4) As much content as possible from copyleft/free sources. Mixamo, Cgtrader, poliigon, textures.com etc. A programmer's job is to use whats out there. But, given an extra 6 months, I could also model and texture everything myself as well. I'll do all level design. 5) Dialogue written by me, voice acting through AI or vocaloid. 6) I would estimate the final code to be around 50,000 lines for everything. I can give you a breakdown if you want. All code is in C99, minspec is a ryzen 5600g, which is like $150 max these days.
Something I've missed? I've described a classic first person adventure shooter like Halo for you there. Simply let me know if I should do something else.
Back in college, we had to write a constructive geometry object renderer that used GLUT simply as a 2D canvas for a scanline-oriented triangle->trapezoid engine that roughly mirrored the capabilities of OpenGL with Phong and Gouraud shading, texture mapping, and bump mapping. It wasn't hard when each piece was broken down and spoon fed (quaternions and transformation matrices). The hardest part was creating a scene of complex object parts to model the real world programmatically.
The problem is, sometimes profilers don't identity anything in particular or unwinding things to a point where they can be fixed is actually very har. A more realistic ROI argument should include this as it is a real problem.
I think code should be reasonably fast out of the box. I'm not suggesting vectorizing everything or even taking extreme steps to avoid allocations, etc. Rather, if an algorithm can easily be O(N), don't dumbly implement an O(N^2) or O(N^24) version. If O(1) is possible, do that unless you know the numbers are going to be very small. Don't make 500 DB calls when one will do and finally, don't program by unexamined phrases - particularly those than include "evil" or "harmful" or other such adjectives.
"A debugger is how you get a view into a system that's too complicated to understand. I mean, anybody that thinks just read the code and think about it, that's an insane statement, you can't even read all the code on a big system. You have to do experiments on the system. And doing that by adding log statements, recompiling and rerunning it, is an incredibly inefficient way of doing it. I mean, yes, you can always get things done, even if you're working with stone knives and bare skins."
1. If I actually pause execution, the thing I'm trying to debug will time out some network service, at which point trying to step forward is only going to hit sad paths
2. The device I'm debugging doesn't *have* a real debugger. (Common on embedded, really common for video games. Ever triggered a breakpoint in a graphics shader?) Here I might substitute "print" for "display anything at all" but it's the same idea really.
Every debugger I've ever worked with has logpoints along with breakpoints that allow you to _dynamically_ insert "print statements" into running code without having to recompile or pollute the code with a line of code you're going to have to remember to remove. So I still think debuggers win.
But... that's where you put breakpoints and then you don't need to "single-step" through code. Takes less time to put a breakpoint then to add (and later remove) temporary print statements.
(Now if you're putting in permanent logging that makes sense, do that anyway. But that probably won't coincide with debugging print statements...)
Printf debugging gives you the full picture of an entire execution at a glance, allowing you to see time as it happened. The debugger restricts you to step through time and hold the evolution of state in your memory in exchange for giving you free access to explore the state at each point.
Occasionally that arbitrary access is useful, but more often than not it's the evolution of state that you're interested in, and printf gives you that for free.
And of course, should the problem you're debugging be throwing exceptions or crashing the app, the debugger can pause the world at that moment for you, and you get the benefit of debugging and having a "printf log" of the same execution already available.
I don't think you can achieve the same result using debuggers, they just stop you there and you have no context how you got there or what happens after.
Maybe some people just aren't good at print debugging, but usually it finds the issue faster for me, since it helps pinpointing where the issue started by giving you a timeline of events.
Edit: And you can see the result of debugger use in this article, under "Expression Complexity" he rewrote the code to be easier to see in a debugger because he wanted to see the past values. That makes the code worse just to fit a debugger, so it also has such problems. When I use a debugger I do the same, it makes the code harder to read but easier to see in a debugger.
One is finding a needle in a haystack - you have no idea when or where the bug occurred. Presumably your logging / error report didn't spit out anything useful, so you're starting from scratch. That and race conditions. Then print statements can be lovely and get you started.
Most of my debugging is a different case where I know about where in code it happened, but not why it happened, and need to know values / state. A few breakpoints before/during/after my suspected code block, add a few watches, and I get all the information I need quite quickly.
Temporary? Nah, you leave them in as debug or trace log statements. It's an investment. Shipping breakpoints to a teammate for a problem you solved three months ago is a tedious and time-consuming task.
Anyway, breakpoints are themselves quite tedious to interact with iterative problem solving. At least if you're familiar with the codebase.
That's not usually the case.
Setting conditional breakpoints, especially for things like break on all exceptions, or when the video buffer has a certain pattern, etc, is usually where the value starts to come in.
I can debug .NET application and change code live, change variable states if needed. Watch variables and all kinds of helpers like stack navigation was immensely improved since I started 15 years ago.
I can say debugging Java/.NET applications is totally different experience than using debugger from 1999.
Multi threaded apps debugging and all kinds of helpers i visual debuggers.
I just fail to see why someone would waste time putting in debug statements when they can configure debug session with conditional break points.
I tend to do both, print statements when I don't feel I want to be stepping through some cumbersome interplay of calls but diving into the debugger to step through the nitty gritty, even better when I can evaluate code in the debugger to understand the state of the data at precise points.
I don't think there's a better or worse version of doing it, you use the tool that is best for what introspection you need.
If you can detect a problematic condition and you want to know what will happen next, a debugger is a great tool.
If you can detect a problematic condition and you need to find out what caused it, it’s printf all the way.
My theory is that different types of programming encounter these two types of problems at different relative rates, and that this explains why many people strongly prefer one over the other but don’t agree on which.
> It takes less time to decide where to put print statements than to single-step to the critical section of code
Why would you ever be single-stepping? Put a break point (conditional if necessary) where you would put the print statement. The difference between a single break point and a print statement is that the break point will allow you to inspect the local variables associated with all calls in the stack trace and evaluate further expressions.
So when do you debug instead of using print statements? When you know that no matter what the outcome of your hypothesis is, that you will need to iteratively inspect details from other points up the stack. That is, when you know, from experience, that you are going to need further print statements but you don't know where they will be.
For example, one thing you wrote that jumps out at me:
> I already have a working model for the way that the code runs [...]
This is not always true. It's only true for code that I wrote or know very well. E.g. as a consultant, I often work on codebases that are new to me, and I do tend to use debuggers there more often than I use print debugging.
Although lots of other variables affect this - how much complicated state there is to get things running, how fast the "system" starts up, what language it's written in and if there are alternatives (in some situations I'll use a Jupyter Notebook for exploring the code, Clojure has its own repl-based way of doing things, etc).
1. Printing vars in unit tests may be the fastest first approach. If i know where the bug may be.
2. When that fails i usually bring in debuggers to unit tests.
3. When these aren't helping, you need debuggers on the entire binary.
4. Still stuck? Use a debugger in production.
3. What if the binary interacts with other networked computers, you gonna debug all of them? Do you end up instrumeting the whole internet? You scope out, you spiral out of control until someone puts a limit.
However, I use prints a lot more because you can, as you say, usually get to the answer faster.
This in no way disagrees with the quote. Both can be true. The quote isn’t saying debuggers can’t provide unique insights, just that for the majority of debugging the print statement is a faster way to get what you need.
I use a debugger when I've constructed a complex process that has a large cardinally of states it could end up in. There is no possibility that I can write logic checks (tests) for all source inputs to that state.
I don't use one when I could simply increase test situations to find my logical error.
Consider the difference between a game engine and a simple state machine. The former can be complex enough to replicate many features of the real world while a simple state machine/lexer probably just needs more tests of each individual state to spot the issue.
Professional debuggers such as the one in IntelliJ IDEA are invaluable regardless of one's familiarity with a given codebase, to say otherwise is utter ignorance. Outside of logging, unless attaching a debugger is impractical, using print statements is at best wasting time.
Than what? In languages with good debugger support (see JVM/Java) it can be far quicker to click a single line to set a breakpoint, hit Debug, the inspect the values or evaluate expressions to get the runtime context you cant get from purely reading code. Print statements require rebuilding code and backing them out, so its hard to imagine that technique being faster.
I do use print debugging for languages with poor IDE/debugger support, but it is one big thing I miss when outside of Java.
It’s impossible? Or would take engineering work to enable
I guess you can attach a debugger for unit tests, but that's not very useful.
That is in fact incredibly useful
It’s a similar muscle to exercise as using a profiler.
But you checking out the code and debugging it means that you are providing the service. Where, exactly, do you find the service (micro or otherwise) boundary in this case?
Or are you just struggling to say that your service utilizes multiple applications?
It also requires that each of your services can scale down as well as they can scale up… none of them should be so huge that your whole cluster can’t fit on a single machine, if you’re just simulating one “request” at a time. (Single instances of everything, don’t reserve memory, etc.) There’s use cases where this isn’t practical, but in most cases it’s very doable.
I don't often use a debugger, and I still feel the need to point out Visual Studio could step into DCOM RPCs across machines in the first release of DCOM, ca. 1995. (The COM specification has a description of how that is accomplished.)
But if there is no support from anyone and you'll be starting from scratch, you've print-debugged and fixed the issue before you get the debugger attached to anything relevant.
I've often felt that I should, but never enough to actually learn how
My workflow is usually:
1. insert a breakpoint on some code that I’m trying to understand
2. attach the debugger and run any tests that I expect to exercise that code
3. walk up and down the call stack, stepping occasionally, reading the code and inspecting the local variables at each level to understand how the hell this thing works and why it’s gone horribly wrong this time.
4. use my new understanding to set new, more relevant breakpoints; repeat 2-4.
Sometimes I fiddle with local variables to force different states and see what happens, but I consider this advanced usage, and anyway it often doesn’t work too well on my current codebase.
One of the first things I do in a codebase is get some working IDE/editor up where I can quickly run the program under a debugger, even if I'm not immediately troubleshooting something. It's never long before I need to use it.
I was baffled when I too encountered this. Even working collaboratively with people they'd have no concept of how to use a debugger.
"No, set a breakpoint there"
"yeah now step into the function and inspect the state of those variables"
"step over that"
: blank stares at each instance :
I don't use debuggers as often in Python, probably because it's eaiser to throw code in a notebook and run line by line to inspect variables, change/inject state and re-run. That's possible but a lot harder to do in C++.
Also for embedded work, using a debugger and memory viewer is pretty powerful. It's not something people think about for Arduino but almost every commodity micro supports some sort of debugwire-like interface (which is usually simpler than JTAG).
I only used debugger recently in C# and C, when I was learning and practicing them.
A point that may be pedantic: I don't add (and then remove) "print" statements. I add logging code, that stays forever. For a major interface, I'll usually start with INFO level debugging, to document function entry/exit, with param values. I add more detailed logging as I start to use the system and find out what needs extra scrutiny. This approach is very easy to get started with and maintain, and provides powerful insight into problems as they arise.
I also put a lot of work into formatting log statements. I once worked on a distributed system, and getting the prefix of each log statement exactly right was very useful -- node id, pid, timestamp, all of it fixed width. I could download logs from across the cluster, sort, and have a single file that interleaved actions from across the cluster.
This is an anti-pattern which results in voluminous log "noise" when the system operates as expected. To the degree that I have personally seen gigabytes per day produced by employing it. It also can litter the solution with transient concerns once thought important and are no longer relevant.
If detailed method invocation history is a requirement, consider using the Writer Monad[0] and only emitting log entries when either an error is detected or in an "unconditionally emit trace logs" environment (such as local unit/integration tests).
0 - https://williamyaoh.com/posts/2020-07-26-deriving-writer-mon...
I usually roll my own wrapper around an existing logging package, but https://www.npmjs.com/package/debug is a good example of what life can be like if you're using JS. Want to debug your rate limiter? Write `DEBUG=app:middleware:rate-limiter npm start` and off you go.
It is an anti-pattern due to what was originally espoused:
I add logging code, that stays forever. For a major
interface, I'll usually start with INFO level debugging, to
document function entry/exit, with param values.
Also, you did not address the non-trivial cost implications of voluminous log output.
> You touch on this, but it's a bit strange to me that the default case is assumed to be "all logs all the time".
Regarding the above, production-ready logging libraries such as Logback[0], log4net[1], log4cpp[2], et al, allow for run-time configuration to determine what "areas" will have their entries emitted. So "all logs all the time" is a non sequitur in this context.
What is relevant is the technique identified of emitting execution context when it matters and not when it doesn't. As to your `npm` example, I believe this falls under the scenario I explicitly identified thusly:
... an "unconditionally emit trace logs" environment
(such as local unit/integration tests).
> the non-trivial cost implications of voluminous log output
If log output is conditional at compile time there are no non-trivial cost implications, and even at runtime the costs are often trivial.
I assumed nothing of the sort. What I did was identify an anti-pattern and describe an alternative which experience has shown to be a better approach.
"Incompetence" is your word, not mine.
> ... and adding the label of "anti-pattern" to something that's evidently got so much nuance to it.
I fail to see the nuance you apparently can see.
>> the non-trivial cost implications of voluminous log output
> If log output is conditional at compile time there are no non-trivial cost implications, and even at runtime the costs are often trivial.
Cloud deployment requires transmission of log entries to one or more log aggregators in order to be known.
By definition, this involves network I/O.
Network communication is orders of magnitude slower than local I/O.
Useless logging of "function entry/exit, with param values" increases pressure on network I/O.
Unless logging is allowed to be lossy, which it never is, transmission must be completed when log buffers are near full capacity.
Provisioning production systems having excessive logging can often require more resources than those which do not excessively log.
Thus disproving:
> ... even at runtime the costs are often trivial.
When considering the implication of voluminous log output in a production environment.
The code that computes data for and assembles your log message may end up executing logic that affects the system elsewhere. If you put that code under conditional, your program will behave differently depending on the logging configuration; if you put it outside, you end up wasting potentially substantial amount of work building log messages that never get used.
As I said, responding to another comment of yours, a distributed system I worked on produced a few GB a day. The logs were rotated daily. They were never transmitted anywhere, during normal operation. When things go wrong, sure, we look at them, and generate even more logging. But that was rare. I cannot stress enough how much of a non-issue log volume was in practice.
So I ask you to quantify: What counts (to you) as voluminous, as in daily log file sizes, and how many times they are sent over the network?
I mean "a lot" or more specifically; "a whole lot."
Here is an exercise which illustrates this. For the purposes here, assume ASCII characters are used for log entries to make the math a bit easier.
Suppose the following:
Each log statement is 100 characters.
Each service invocation emits 50 log statements.
Average transactions per second during high usage is 200tps.
High usage is on average 2 hours per day.
100 x 50 x 200 x 60 x 60 x 2 = 7_200_000_000 = 7.2GB / day
The quantification is above and regarding log entries being sent over a network - in many production systems, log entries are unconditionally sent to a log aggregator and never stored in a local file system.
But in normal usage, even in the scenario you describe your argument about log volume is not convincing.
Resorting to an ad hominem and non sequiturs are we? Really?
C'mon, you are better than that.
Well, I agree completely, but those conditions are a tall order. The whole point of debugging (by whatever means you prefer) is for those situations in which things don't succeed or operate as intended. If I have a failure, and suspect a major subsystem, I sure do want to see all calls and param values leading up to a failure.
In addition to this point, you have constructed a strawman in which logging is on all the time. Have you ever looked at syslog? On my desktop Linux system, output there counts as voluminous. It isn't so much space, or so CPU-intensive that I would consider disabling syslog output (even if I could).
The large distributed system I worked on would produce a few GB per day, and the logs were rotated. A complete non-issue. And for the rare times that something did fail, we could turn up logging with precision and get useful information.
> Well, I agree completely, but those conditions are a tall order.
Every successful service invocation satisfies "all collaborations succeed and the system operates as intended." Another way to state this is every HTTP `1xx`, `2xx`, and `3xx` response code produced by an HTTP service qualifies as such.
> The whole point of debugging (by whatever means you prefer) is for those situations in which things don't succeed or operate as intended.
Providing sufficient context in the presence of errors, or "situations in which things don't succeed or operate as intended", was addressed thusly:
If detailed method invocation history is a requirement,
consider using the Writer Monad and only emitting log
entries when either an error is detected or in an
"unconditionally emit trace logs" environment (such as
local unit/integration tests).
See above.
> In addition to this point, you have constructed a strawman in which logging is on all the time.
No, I addressed your original premise in the context of a production web application, where logging is configured during deployment.
See also your own contradiction by previously asserting, "I sure do want to see all calls and param values leading up to a failure."
So which is it?
Did I construct a strawman "in which logging is on all the time"?
Or do you "want to see all calls and param values leading up to a failure", which requires "logging is on all the time"?
> Have you ever looked at syslog?
This is a strawman. Syslog is a component for logging and has nothing to do with the programs which use it.
> The large distributed system I worked on would produce a few GB per day, and the logs were rotated. A complete non-issue.
If this is the same system you described in a different comment also in this thread, I identified a standard industry practice of log entries produced by application nodes in a production environment being unconditionally sent to a log aggregator and not stored in a local file system. The reasons for this are well documented.
I don't know from Writer Monads. But if you only emitting log entries on some future failure or request, then that's potentially a lot of logging to keep somewhere. Where? Log aggregator? Local files? Memory? What about log volume? Does this writer monad implement log rotation? It sounds like you are sweeping a lot of the things you object to under this writer monad rug.
Let me be real clear about all calls and param values leading up to a failure.
- In normal operation, turn logging off completely, or turn on some level that produces tolerable log volume, (it seems like your threshold is much lower than mine).
- When a failure occurs: Restart the service with more logging enabled, (hence the all calls an param values), so that you have logging when the failure occurs again.
About local logs vs a log aggregator: The system I worked on was a shared nothing archive. To add storage and metadata capacity, you add nodes. Each node also stored its own log files. I get that this may not be the answer all the time, and that a log aggregator is useful in some scenarios. However, even in that case, your concerns about log volume seem overblown to me.
My point was to identify how common the "happy path" scenario is and was in direct response to:
>> There is no value for logging "function entry/exit, with param values" when all collaborations succeed and the system operates as intended.
> Well, I agree completely, but those conditions are a tall order.
Regarding your question:
> What if the HTTP server crashes, and doesn't send a response at all?
Again, the HTTP status codes were used to illustrate the frequency of successful invocations. But to your point, if an HTTP server crashes then log entries for in-flight workflows would likely not be emitted. A similar possibility also holds for local file system logging as buffering exists (sometimes on multiple levels).
> I don't know from Writer Monads.
No worries. All it is is a formal definition of a type having specific capabilities called "laws" in the functional programming world.
> But if you only emitting log entries on some future failure or request, then that's potentially a lot of logging to keep somewhere. Where? Log aggregator? Local files? Memory?
What is meant by "future failure" is a failure potentially encountered during the evaluation of a single invocation. In the context of a HTTP server, this would be the handling of each submission to a specific HTTP endpoint and verb. This is often defined within an IO Monad[0], but does not have to be, and is out of scope for this discussion.
The answer to the rest of your questions is the deferred log entry definitions are held in memory for the duration of the single service invocation, with any log emissions produced transmitted to a network-accesible log aggregator via the logging component used.
> Let me be real clear about all calls and param values leading up to a failure.
The scenario you kindly shared is understandable, yet is one which has been unacceptable in teams I have worked with. Losing the "first error" is not an option in those environments.
0 - https://en.wikibooks.org/wiki/Haskell/Understanding_monads/I...
What you describe is leaving around hand-rolled instrumentation code that conditionally executes expensive reporting actions, which you can toggle on demand between executions. Thing is, this is already all done automatically for you[1] - all you need is the right build flag to prevent optimizing away information about function boundaries, and then you can easily add and remove such instrumentation code on the fly with a debugger.
I mean, tracing function entry and exit with params is pretty much the main task of a debugger. In some way, it's silly that we end up duplicating this by hand in our own projects. But it goes beyond that; a lot of logging and tracing I see is basically hand-rolling an ad hoc, informally-specified, bug-ridden, slow implementation of 5% of GDB.
Why not accept you need instrumentation in production too, and run everything in a lightweight, non-interactive debugging session? It's literally the same thing, just done better, and couple layers of abstraction below your own code, so it's more efficient too.
--
[0] - https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule
[1] - Well, at least in most languages used on the backend, it is. I'm not sure how debugging works in Node at the JS level, if it exists at all.
I think there's still value in adding toggleable debug output to major interfaces. It tells you exactly what and where the important events are happening, so that you don't need to work out where to stick your breakpoints.
I don't want logging done automatically for me, what I want is too idiosyncratic. While I will log every call on major interfaces, I do want to control exactly what is printed. Maybe some parameter values are not of interest. Maybe I want special formatting. Maybe I want the same log line to include something computed inside the function. Also, most of my logging is not on entry/exit. It's deeper down, to look at very specific things.
Look, I do not want a debugger, except for tiny programs, or debugging unit tests. In a system with lots of processes, running on lots of nodes, if a debugger is even possible to use, it is just too much of a PITA, and provides far too miniscule a view of things. I don't want to deal with running to just before the failure, repeatedly, resetting the environment on each attempt, blah, blah, blah. It's a ridiculous way to debug a large and complex system.
What a debugger can do, that is harder with logging, is to explore arbitrary code. If I chase a problem into a part of my system that doesn't have logging, okay, I add some logging, and keep it there. That's a good investment in the future. (This new logging is probably at a detailed level like DEBUG, and therefore only used on demand. Obvious, but it seems like a necessary thing to point out in this conversation.)
The problem is that if you only log problems or "important" things, then you have a selection bias in the log and don't have a reference of how the log looks like when the system operates normally.
This is useful when you encounter unknown problem and need to find unusual stuff in the logs. This unusual stuff is not always an error state, it might be some aggregate problem (something is called too many times, something is happening in problematic order, etc.)
A case can be made for only logging the steps performed up to and including an error. This even excludes logging "important things" other than those satisfying system/functional requirements (such as request/response audit logs).
It is reminiscent of "the Unix philosophy", but different in important ways, and is essentially:
Capture what would be log entries if a there is a
future unrecoverable error.
If an error is encountered, emit *all* log entries starting
from the earliest point (such as receiving an external
event or a REST endpoint request) up to and including the
information detailing the unrecoverable error.
If the workflow succeeds, including producing an expected
failed workflow response (such as a validation error),
discard the deferred log entries.
While using a functional programming language and employing referentially transparent[0] abstractions (such as the Writer Monad) usually makes implementing this pattern much simpler than when using an imperative language, it can be successfully done with the latter given sufficient discipline and the workflow implementation being referentially transparent[0].
An important complement to the above is to employ other industry standard verification activities, such as unit/feature/integration tests.
Some IDEs promise to solve that, but I’ve not been impressed thus far.
YMMV based on language/runtime/toolkit of course. This might get added to my wishlist for my next language of choice.
I can always drop an entire state object into the log if I need it, but the only way for a debugger to approximate what a log can give me is for me to step through a bunch of break points and hold the time stream in my head.
The one place where a debugger is straight up better is if I know exactly which unit of code is failing and that unit has complicated logic that is worth stepping through line by line. That's what they were designed for, and they're very useful for that, but it's also not the most common kind of troubleshooting I run into.
- Log4j + MDC's so we could get a timestamp and thread ID easily
- some python scripting
I can't remember if it was PDF or SVG, but I do remember having to choose something that would allow for huge graphs and lots of scrolling!
These days there's better options on github, but nice to see I was on a decent path.
You can script this so all function entry/exists, or whatever, are logged without touching the code or needing to recompile. You can then bulk toggle these breakpoints, at runtime, so you only see a particular subset when things get interesting.
Modifying the code to print stuff will feel barbaric after driving around a fine tuned debugging experience.
Print debugging, while not clever or powerful, has never once failed me.
You can attach lldb without Xcode. Or you can open the lldb terminal in Xcode, pause execution, and inspect the breakpoints manually
It's one point in favor of async, and language that make async debugging easy.
And at the very least, you can encapsulate things so they can be debugged separately from the multi threading stuff.
If you're always using the debugger, you learn to build around it as one of your primary tools for interacting with code, and you notice right away if the debugger can't help with something.
Wtaf?
If there isn't one, I'd rather use a language with a debugger and write a thousand lines of code than 100 lines of code in a language I'm going to have to black box.
What sold me on debugger is the things you can do with it
* See values and eval expressions in calling frames.
* Modify the course of execution by eval'ing a mutating expression.
* Set exception breakpoints which stop deep where the exception is raised.
Not any excuse, but another factor to be considered when adding infra layers between the developer and the application.
Debuggers actually can hide entire categories of bugs caused by race conditions when breakpoints cause async functions to resolve in a different order than they would when running in realtime
I've never worked somewhere where logging is taken seriously. Like, our AWS systems produce logs and they get collected somewhere, but none of our code ever does any serious logging.
If people like print-statement debugging so much, then double down on it and do it right, with a proper logging framework and putting quality debug statements into all code.
That's all event-sourcing is.
By all means, learn to use a debugger well but don't become overly dependent on them.
If you usually aren't able/allowed to use a debugger in production and must rely on observability tools, it's helpful to know how to utilize those tools locally as effectively as possible when debugging.
I really don't get this at all. To me it is infinitely easier to iterate and narrow the problem rather than trying to identify sight-unseen where the problem is—it's incredibly rare that the bug immediately crashes the program. And you can fit a far higher density of relevant information through print statements over execution of a reproduced bug than you can reproduce at any single point in the call stack. And 99% of the information you can access at any single point in the call stack will be irrelevant.
To be sure, a debugger is an incredibly useful and irreplaceable tool.... but it's far too slow and buggy to rely on for daily debugging (unless, as you indicate, you don't know the codebase well enough to reason about it by reading the code).
Things that complicate this:
* highly mutable state
* extremely complex control or data flow
* not being able to access logs
* the compiler lying to you or being incorrect
* subtle instruction ordering issues
i'd rather never use debugger, so that my coding style is enforced to be clean code, strong type system, immutable variables, explicit error control, explicit control flow etc
In short, which is better? It depends, and varies wildly by domain.
The only thing that's really missing is history of the values of the watches you added, all in one log. With filters because why not.
For some reason I never seen this option in any debugger I tried.
Heh, who knew, debuggers already have this feature under the name tracepoints (VS) or logpoints (Viscose).
https://code.visualstudio.com/blogs/2018/07/12/introducing-l...
It should be way more well known among print debug crowd.
TIL Linus Torvald is a young grug.
Now, some 35 years later, I greatly prefer logs. This way I can compare execution paths of different use cases, I can compare outcomes of my changes, etc. I am not confined to a single point of time with tricky manual control as with debugger, I can see them all.
To old grugs: learning to execute and rewind code in your head is a major superpower. And it works on any codebase.
However debugging is an essential tool in the arsenal. If something is behaving oddly even the best REPL can't match debugging as a dev loop (maybe lisp excepted).
I even miss the ability to move the current execution point back in .NET now I use Go and JS. That is a killer feature. Edit and continue even more so!
Then next level is debugging unit tests. Saved me hours.
Which is a good thing! Easily comprehended code that you can reason about without stepping through it is good for grug brain.
But in 2000 I started working in a role which required understanding the misbehavior of embedded systems that were forwarding the live traffic, there was a technical possibility to do “target remote …” but almost never an option to stop a box that is forwarding the traffic.
So you end up being dependent on debugs - and very occasional debug images with enhanced artisanal diagnostics code (the most fun was using gcc’s -finstrument-function to catch a memory corruption of an IPSec field by an unrelated IKE code in a use-after free scenario)
Where the GDB shined though is the analysis of the crash dumps.
Implementing a “fake” gdb stub in Perl, which was sucking in the crash dump data and allow to leisurely explore it with debugger rather than decoding hex by hand, was a huge productivity boon.
So I would say - it’s better to have more than one tool in the toolbox and use the most appropriate one.
Your experience sounds more sophisticated than mine, but the one time I was able to get even basic debugger support into a production Ruby app, it made fixing certain classes of bug absolutely trivial compared to what it would have been.
The main challenge was getting this considered as a requirement up front rather than after the fact.
Sigh. Same. To a large extent, this is caused by debuggers just sucking for async/await code. And just sucking in general for webdev.
On the other hand, the humble console.log() just works without requiring insanely tedious and frustrating debugger steps.
Line debuggers usually have some nice conveniences, but the major bottlenecks are between the ears, not in the tool.
So long as no one sees your workflow as valuable, they will happily destroy it if it means getting the ticket done.
I used to debug occasionally but haven't touched a debugger in years. I'm not sure exactly why this is, but I'm generally not concerned with exactly what's in the stack on a particular run, but more with some specific state and where something changes time after time, and it's easier to collect that data programatically in the program than to figure out how to do it in the debugger that's integrated with whatever IDE I'm using that day.
And the codebase I deal with at work is Spring Boot. 90% of the stack is meaningless to me anyway. The debugger has been handy for finding out "what the f*$% is the caller doing to the transaction context" but that kind of thing is frustrating no matter what.
Anyway, I think they're both valid ways to explore the runtime characteristics of your program. Depends on what you're comfortable with, how you think of the problem, and the type of codebase you're working on. I could see living in the debugger being more useful to me if I'm in some C++ codebase where every line potentially has implicit destructor calls or something.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
Another way (and probably a better idea) is creating a launch definition for VS Code in launch.json which attaches the IDE's debugger to Chrome. Here is an article describing how that works: https://profy.dev/article/debug-react-vscode
Breakpoints are nice because they can't accidentally get shipped to production like debugger calls can.
For Python I essentially do the same thing, minus involving Chrome. I run the entry point to my application from launch.json, and breakpoints in the IDE 'just work'. From there, you can experiment with how the debugger tools work, observe how state change as the application runs, etc.
If you don't use VS Code, these conventions are similar in other IDEs as well.
Even assuming print statements and debuggers are equally effective (they're not), debuggers are better simply because they are faster. With print statements, you might need to recompile a dozen times before you find whatever it is you're looking for. Even with quick builds this is infuriating and a ton of wasted time.
The trouble of course is that there is always money to be made debugging. There is almost no incentive in industry to truly eliminate bugs and, indeed, I would argue that the incentives in the industry actively encourage bugs because they lead to lucrative support contracts and large dev teams that spend half their time chasing down bugs in a never-ending cycle. If a company actually shipped perfect software, how could they keep extracting more money from their customer?
Built code largely works with source maps, but it fails often enough, and in bizarre ways, that my workflow has simply gone back to console logs.
React's frequent re-renders have also made breakpoints very unpleasant - I'd rather just look at the results of console logs.
Are there ways I can learn to continue enjoying the debugger with TS+React? It is still occasionally useful and I'm glad its there, but I have reverted to defaulting to console logs.
But yes, any code that is inside jsx generally sucks to debug with standard tooling. There are browser plugins that help you inspect the react ui tree though.
Conditional breakpoints help alleviate the pain when there are frequent re-renders. Generally you can pinpoint a specific value that you're looking for and only pause when that condition is satisfied. Setting watch expressions helps a lot too.
But the reason is simple: they are always 100x harder to set up and keep working than just type a print statement and be done with it.
Especially when you're in a typical situation where an app has a big convoluted docker based setup, and with umpteen packers & compilers & transpilers etc.
This is why all software companies should have at least one person tasked with making sure there are working debuggers everywhere, and that everyone's been trained how to use them.
There should also be some kind of automated testing that catches any failures in debugging tooling.
But that effort's hard to link directly to shipped features, so if you started doing that, management would home in on that dev like a laser guided missile and task him to something with "business value", thereby wasting far more dev time and losing far more business value.
I’ve spent time at small startups and on “elite” big tech teams, and I’m usually the only one on my team using a debugger. Almost everyone in the real world (at least in web tech) seems to do print statement debugging. I have tried and failed to get others interested in using my workflow.
I generally agree that it’s the best way to start understanding a system. Breaking on an interesting line of code during a test run and studying the call stack that got me there is infinitely easier than trying to run the code forwards in my head.
Young grugs: learning this skill is a minor superpower. Take the time to get it working on your codebase, if you can.