"In fact, how people can debug large existing codebases without a debugger is beyond me."

I currently work on a product with a 15-20 Mloc codebase (mostly C). The development facets which give the closest parallel to a line-by-line debugger are:

Components in the system communicate with each other by sending signals through the framework. All such signals are traceable in all builds, and can easily be decoded. These narrow down which components is doing something wrong, and what exactly it's black-box behaviour is (i.e. when I see bug B, it's because component C is giving output O after receiving input I).

Lots of trace. All branches are traced in debug builds, unless there's a serious perf reason not to. Many interesting variables are traced too.

Extensive UT and FV, using powerful (rubbish in other ways) frameworks. Any real-world bug should be reproducible in automated tests, which you can build and run with debug-level trace.

It's not always fast, but with these you can always find the problem line / section with typical code bugs. Obviously some types of bug (e.g. perf) often don't come down to 'a line of code' or 'a bad interaction', so aren't necessarily resolved this way, but that's the same with line-by-line debuggers, in my experience.

Tracing is great but often gives you a low signal-to-noise ratio and for new code, is often missing for the bug you're trying to investigate.

A debugger lets you add tracing as the system is executing, which in my experience is extremely productive, especially in C++ codebases where adding some logging can result in 10 minutes to rebuild and deploy. And I don't know what I would do without data breakpoints that give you the callstack and threading info when an address is written to.

In case it wasn't clear: I don't mean to suggest that no-debugger is superior to debugger.

WRT the benefits of interactive debuggers, I definitely see the advantages of faster debug cycles. Luckily, it's not often that I need to worry about the 'often missing for the bug' / 'callstack and threading info when an address is written to' style of bug by virtue of working in codebases with strong coding standards and diligent code review. Most of the bugs I need to fix are 'developer didn't think of scenario S when writing feature F' or 'developer has forgotten edge-case E', rather than 'code is broken e.g. address re-use'.

And as I said in a cousin comment, I can't deploy a system with trace. That 10 minutes is just spent building :-p

What do you use for tracing?

Nothing special - just macros (provided by the framework that the components run in), which allow the programmer to filter based on filename and/or log level, and print the filename, function and text to file.

There's so much trace that you can't use it in production mode, even if you have a test system which isn't handling any load, so it's purely a debug tool.

> There is simply nothing that comes close to learning and debugging code line-by-line, examining variables and taking branches in real time. In fact, how people can debug large existing codebases without a debugger is beyond me.

Debuggers as a general purpose debugging tool never scaled up for me. Working out a subtle data flow or logic bug in a small piece of code, or having a post-mortem debugger in a failed test, sure, but working with a larger codebase was always very cumbersome for me.

For most bugs I fix I never fire up a a debugger. Often the bug is obvious from the trace, or even just the line number of a failed assertion. If the bug is not obvious at this point I tend to do some manual analysis and looking at the code and flow. By now >98 % are fixed. Often it's more difficult and time-consuming to write the regression test case than the actual debugging.

Sometimes there is that rare elusive bug. But I find that thinking about it and the code works better than using a line debugger, especially for larger systems.

I also find that many "classic" tools are highly inadequate to debug distributed, threaded or asynchronous (or a combination thereof!) software -- similar to how a lot of tooling just runs into a brick wall with threaded software. (Luckily Intel VTune exists).

>Of course, nowadays in modern webdev it seems that most developers no longer use step-debuggers

You make it sound like step-debuggers were the traditional, established thing.

Whereas most programmers back in the day would get by with some carefully placed print statements (based on specific intuition) instead of stepping around to see what goes on.

"The most effective debugging tool is still careful thought, coupled with judiciously placed print statements." (Brian Kernighan).

I've used both, and still do use both in concert with each other.

The debugger is a tool for gaining that intuition about how the program operates, and carefully-placed log statements are a tool for cutting the search space of where you need to place breakpoints and single-step.

"carefully-placed log statements are a tool for cutting the search space!

Indeed, inevitably serious problems tend to be where there is no or poor logging :-)

I'm a heavy user of the excellent Visual Studio debugger for C++ and C#. But for years I didn't use an IDE or debugger for server side Python; I did a lot of Zope & Django debugging with print statements and log files. A couple of years ago I started using IDEA's PyCharm. It's massively accelerated how quickly I can fix bugs in Tornado based server code. More recently I've switched my server side coding style to use coroutines, and that requires a shift in debugging mindset as one can't make the same control flow assumptions as for conventional callback style single threaded async code. I'm looking forward to debuggers that understand how to step over yield or await.

Oh yes...don't get me wrong, there will always be room for crash-and-burn debugging (my pet name for that style), and no doubt interpreted languages make it a much faster and safer method than doing the same thing in C back in the day.

Of course memory corruption was always a nasty potential culprit in C, and once again, using a step-debugger help with those issues immensely for a variety of obvious reasons as you could very quickly often see the exact line the program blew up.

BK's explicit endorsement of C&B debugging was bound to show up sooner or later, and we didn't even really touch on the difficulty of using a stepper in multi-threaded network apps, but remember I simply posited that stepping through code in real-time helped me quickly learn from devs much smarter and more experienced than I.

For me that only applied to ZX Spectrum coding, ever since I moved into MS-DOS and also when coding with friends on their Amiga systems, that I have always used a debugger.

Only when the environment forbids me of using one, do I make use of such techniques.

> In fact, how people can debug large existing codebases without a debugger is beyond me.

Tracing and telemetry. Debuggers are very useful, but only when they're needed. In fact, in large distributed systems 'debugging' end-to-end bugs is usually hindered with a debugger.

As an aside, even when I have to use a debugger during runtime, the largest part of my interaction with it is heavily scripted, and the debugging process is not interactive.

Large existing codebases are always a handful. But it's the small embedded environments where you might not have a debugger that I've experienced. Or the problem is inherently realtime so cannot be slowed down sensibly. For example when debugging a USB endpoint, it would be nice to stop and examine a packet, but not responding in time causes the host to un-enumerate the device.

Twice I've had to use the technique of "write values continuously to uninitialized SRAM, then on reboot print out what you find there" to investigate this kind of thing.

The worst I ever had to do was on an embedded machine with no CPU cache. We were getting a crash, but we couldn't figure out how the code was getting into that function. So we hooked up a logic analyzer to the address bus (this was years ago, when CPUs were DIP packages). We wrote to an unused address decode when we entered the function, used that decode to trigger the logic analyzer, and were able to read back what addresses the CPU had been executing before it entered the function. With that information, everything became clear.

Debuggers like Visual Studio, windbg, gdb are excellent tools but they can't always take you to the problem, especially for multi-threaded server code. Yes, they're great for bringing your mental model of the code execution into line with the real behaviour of the code. But for nailing tricky sunchronisation issues in multi-threaded server code you'll need plenty of configurable logging, and memory profilers like valgrind or purify.

Interesting point. IDEA supposedly added better support for "async stacktraces" recently-- https://blog.jetbrains.com/idea/2017/02/intellij-idea-2017-1... but I'm yet to give it a shot.

Would be good to see this in PyCharm to enable stepping over yield in @tornado.gen.coroutine methods. The whole step over step in debugging model assumes traditional imperative code and stack frame continuity.

I do a lot of back-end web service work and I really enjoy using unit test debugging, it's really fast and gives you a good clean starting point. The times that I have to hook up the debugger to the web server to debug against the website in the browser can be really frustrating and a lot more work especially since the client-side code will call out again to retry web api calls that took too long. I'm yet to find an effective way to debug in this way without deleting breakpoints that get hit and then assigning new ones (and who knows which call exactly I'm debugging but since the data is the same it usually doesn't matter). Does anyone have a better way to debug than the way I'm describing, I use Visual Studio and haven't found a way to easily limit it to just the first call attempt?

I agree - debugging is an integral part of programming. I don't mind debugging, and often bugs are good for you:

- solving a bug teaches you something. You either learn something new about how to write good code, about the product or about the domain.

- debugging problems teaches you how to write code that is easy/easier to debug. You see what information you wish you have had, how to be able to track what is happening in the program etc.

- solving problems and mysteries (how can this even happen) is often fun in itself (like solving sudoku or crosswords or reading murder mysteries).

- when you resolve a bug, both you and your customer are happy. It's a bit of a paradox. If there wasn’t a bug in the first place, there wouldn’t be a need to fix it, so why should they be happy? However, my experience is that they are happy to receive a bug-fix, especially if it is solved quickly

More detailed argument here: https://henrikwarne.com/2012/10/21/4-reasons-why-bugs-are-go...

"debugging is an integral part of programming"

Can't remember where I read this: programming is 20% writing bugs, and 80% fixing them.

I recently fixed a bug in fontforge to do with extracting fonts when parsing pdfs. It took me a whole day because I didn't know what I was doing (haven't worked with C since Uni over 15 years ago).

Because I didn't think to look into using a debugger, I did it very much the hard way by adding trace statements all over the place to slowly narrow down the code path. I knew I was looking for something like that looked like an iteration typo, but it was a large unfamiliar codebase so it tok a long time.

Got there in the end! Probably 2 days of work start to finish for a 1 character fix. Classic.

I will definitely be starting by re-learning to use a step debugger in C next time I'm in the same position.

https://github.com/fontforge/fontforge/pull/3050/commits/e4b...

> In fact, how people can debug large existing codebases without a debugger is beyond me.

Some people work with systems that operate in real time (networked systems are usually in this category) or otherwise can't be instrumented (embedded systems?) and cannot be replayed at will. Not everything is a desktop application or simple CRUD.

Web developers​ use debuggers too.