Can you no longer use windbg to attach to a running kernel on a remote box, and debug the remote box from your development machine? I believe that's how kernel debugging is generally done.

(I’m a Linux kernel dev)

Tools like windbg probably work great until you are debugging the code that runs when windbg tries to take over. Or you’re debugging something sensitive to interrupts and it’s literally impossible to keep up with a tool like windbg. Or you a debugging something that overwrites windbg in memory because you have a corrupt pointer. Or you triple-fault the machine and it reboots with so much prejudice that windbg doesn’t have a chance. Or you’re poking at hardware that the debugger can’t usefully interact with. Etc.

Basically, in the kernel, a lot of the things that a working kernel does to hold your hand aren’t available. So there’s a lot of time spent just thinking or adding logging statements or otherwise using low tech tools. (Or using high tech instrumentation or sanitizers!)

Kernel development is great :)

I have debugged the kernels of Windows, Linux and now iOS using internal tools. Windbg generally works, and is far superior to the equivalent tools for other kernels. Like vim it has a baroque syntax and a steep learning curve, but it also has a gui with big buttons to compensate. Yeah, you can get yourself in bad situations, especially in early boot, but I miss it.

I confess that it’s been years since I used WinDbg. It was quite bare bones back then.

These days I mostly use gdb (which is incredibly buggy for such things) attached to QEMU’s gdbserver. It works except when I doesn’t.

I thought everyone doing OS/kernel dev would be running code in a VM they can pause, snapshot, and single-step all without a guest debugger?

Virtual machines are used extensively by kernel developers but are not an adequate substitute for real hardware in a lot of situations. After all, one of a kernel's jobs is to abstract away differences in underlying hardware.

Windows kernel can generate ETW profiling data, no need to manually add logging information after the fact.

https://docs.microsoft.com/en-us/windows-hardware/drivers/de...

Having worked on both windows and Linux kernel development, I can easily say that Windows has superior debugging facilities. It's pretty rare to make the debugger fall over. Working in the kernel is certainly not without its own challenges, but I feel like Linux makes it harder than it needs to be.

Does x86 have a JTAG like / hardware debugger?

You can get something like JTAG from Intel with a NDA.

https://designintools.intel.com/Silicon_View_Technology_Clos...

Most of the time you can do kernel dev with a serial port and printf though.

That’s probably true, but sometimes you have to go one level deeper right? What happens when your debugger has crashed?

In 8 years on the Windows team I can't recall the debugger ever crashing. If it does, you open the debugger again and reconnect to the target machine.

Heh, I wish that had ever been the case for me. Sometimes it feels like WinDbg freezes every other command. Plus, getting it to reconnect to a kd after it freezes and I kill it was such an exercise in frustration that if the target machine were my own repro box instead of a stress break remote, I would have just hard-reset the machine or (if it were a VM) restored it from snapshot.

> > 3. Designing systems in general. Some people love the challenge of distributed transactions, eventual consistency and all that jazz, but it just rubs my brain the wrong way. I'm not interested at all in that problem space [1].

>I'm not sure I understand this issue. I mean, kernels have subsystems for doing stuff; you might need to design one someday? But it won't be dull-as-dishwater web technology stacks, it'll be you writing data structures directly in C/C++ or Rust if MS goes there.

Ah, I meant designing distributed systems. Like, figuring out the difference services, storage requirements, databases, then planning out the infrastructure. It's really not my cup of tea. I do enjoy designing systems/subsystems/components as long as they're within a single host :)

  Serial printf logging is often required

This reminded me using "printk" in an incorrect place in an operating system course assignment. It was actually printk itself that was hanging the kernel!