* Run the whole tree of processes under `gdb` with `set detach-on-fork off`.
* LD_PRELOAD a library that inserts the sleeps for you, maybe on startup or maybe on signal/exit.
Ideally, we'd have some kind of infrastructure to name and identify particular processes recursively.
This is one area where rust disappoints me, there isn’t a “cargo debug” built in (there is an external program but it doesn’t work well), and when I just manually attach gdb most of the symbols are usually missing.
I would seriously consider a language billed as “debugger-first”, just to see what the experience was like.
Use C, tell gdb "-O0 -g -ggdb3"
I still sometimes get caught out getting debug information all in the right place, particularly when someone is using ninja or some such, I've even ended up wrapping gcc just to strip optimisation options and add -g, rather than figure out how to fight some very complex build system mess.
java -jar -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005
If your app is a gradle app, the flag is "--debug-jvm"Then you connect to it with "jdb"
I don't think I want to do it myself, but I'd love a "101 ways to get into a debugger" webpage.
When it is loaded it will automatically talk to VSCode and tell it to start a debugger and attach to it & it waits for the debugger to attach.
End result is you just have to run your script with an environment variable set and it will automatically attach a nice GUI debugger to the process no matter how deeply buried in scripts and Makefiles it is.
https://github.com/Timmmm/autodebug
I currently use this for debugging C++ libraries that are dynamically loaded into Questa (a commercial SystemVerilog simulator) that is started by a Python script running in some custom build system.
In the past I used it to debug Python code running in an interpreter launched by a C library loaded by Questa started by a Makefile started by a different Python interpreter that was launched by another Makefile. Yeah. It wasn't the only reason by a long shot but that company did not survive...
System.Diagnostics.Debugger.Launch();
Which pops up a window asking you to select what debugger you want to use, and then opens your application in it.
We (Undo.io) came up with a technique for following a tree of processes and initiating process recording based on a glob of program name. It's the `--record-on` flag in https://docs.undo.io/UsingTheLiveRecorderTool.html. You can grab a free trial from our website.
For open source, with rr (https://rr-project.org/) I think you'd just `rr record` the initial process and you'll end up capturing the whole process tree - then you can look at the one you're interested in.
As others have said you could also do some smart things with GDB's follow-fork settings but I think process recording is ideal for capturing complicated situations like this as you can go and review what happened later on.
> In the bootstrap process, the entire thing becomes way more complex. You see, rustc is not invoked directly. The bootstrap script calls a wrapper around the compiler.
> Running that wrapped rustc is not easy to run either: it requires a whole lot of complex, environment flags to be set.
> All that is to say: I don’t know how to debug the Rust compiler. I am 99.9 % sure there is an easy way to do this, documented somewhere I did not think to look. After I post this, somebody will tell me "oh, you just need to do X".
> Still, at the time of writing, I did not know how to do this.
> So, can we attach gdb to the running process? Nope, it crashes way to quickly for that.
It's kind of funny how often this problem crops up and the variety of tricks I have in my back to deal with it. Sometimes I patch the script to invoke gdb --args [the original command] instead, but this is only really worthwhile if it's a simple shell script and also I can track where stdin/stdout are going. Otherwise I might patch the code to sleep a bit before actually running anything to give me a chance to attach GDB. On some platforms you can get notified of process execs and sometimes even intercept that (e.g. as an EDR solution) and sometimes I will use that to suspend the process before it gets a chance to launch. But I kind of wish there was a better way to do this in general…LLDB has a "wait for launch" flag but it just spins in a loop waiting for new processes and it can't catch anything that dies too early.