This is an approach I use for reproducing issues on userspace applications as well. I happen to maintain a fairly large Windows-based application at my workplace, and sometimes the steps to reproduce an issue could take minutes to hours just to get you to the point of being able to try and replicate the issue. I personally use VMware Workstation at work, and since I'm on a quick SSD, taking and reverting multiple snapshots has little performance cost.

I've always thought it would be nice if the kernel exposed this ability to regular userland applications. On Unix systems, Ctrl+Z of a terminal application will suspend it, but not across reboots. Core files are very close, yet typically they are only produced in response to a crash, and aren't really meant for resuming a process. Obviously there's lots of limitations (eg shared memory could cause issues, hardware states could change), but for a lot of cases it would probably work pretty well.

Not only is it great for triggering crashes, but if you then broke out into a debugger, you could toy with toggling register and memory contents and then single-step through the instructions to see what happens. It's really substantially easier to debug software under emulation because of features like these.

Sounds like you're looking for the checkpoint/restore functionality that has been in the works for a few years now. Check out http://en.wikipedia.org/wiki/CRIU for more details, seems they have things mostly working now.

DragonFly BSD does this. They call it process checkpointing.

Processes under DragonFly may be "checkpointed" or suspended to disk at any time. They may later be resumed on the originating system, or another system by "thawing" them.

http://www.dragonflybsd.org/features/#index8h2

OpenBSD and FreeBSD call it suspend/resume I think:

http://openbsd.org/papers/zzz.pdf

http://wiki.freebsd.org/SuspendResume

Those links are talking about suspend/resume of the computer, which is a totally different thing.

You might enjoy rr, 'record-replay', which works kind of like gdb (and I believe is based on it) after you record a program execution --- you can then use gdb multiple times on the stream of execution. rr-project.org.

Honest question: what guarantees that if the bug reproduced 10 minutes after a snapshot, it will reproduce again if you resume running nondeterminstically after that snapshot ?

That's always an issue, but in my testing with emulators, once you rule out external inputs (eg keyboard input), it tends to become predictable and reproducible. And using a VM should help dramatically in reducing randomness. And you keep capturing states and narrowing the window until you can reproduce it in eg ten seconds; as too much can happen in the span of ten minutes. The trick is that you have to get a state right before the problem actually begins, but you don't yet know where that is. So it's possible your state capture may be a bit too late, after the issue already eg corrupted memory somewhere.

I know that there are certainly bugs that this kind of technique would never work on. I've hit a few bugs that could only be triggered on live hardware before. But I'm curious if they've tried this kind of approach for this bug yet or not.

Could you post more about your technique? Like, what emulator are you using (qemu)? How do you trigger the snapshots? Are you snapshotting memory or disk or both? How much disk space is consumed by all these snapshots? Do you discard snapshots? How do you know when the bug has been triggered?

Sure, it's my own software. I hook up saving and loading states to key presses, so eg F5 would save, F7 would load. And then F6/F8 would increment or decrement the save slot number.

You basically have to capture everything possible: all memory, the state of all the CPU registers, the state of all hardware registers, etc. Obviously disk would be a real challenge, where you'd have to keep a delta list of disk changes since program start, or simply not serialize that state. If you miss anything, you can have problems loading states correctly. However, there is quite a bit of tolerance between theory and practice, so if there is something that you really can't capture the state of for some reason (like a hardware write-only register when you weren't logging what was previously written to it), a lot of times you can get away with it anyway.

Because the system I am using is so old, snapshots are only 300KB each. Sometimes I dump them to disk, sometimes I just keep them in RAM. I know that a PC would be much more challenging, given how much more hardware is at play, and because VMs aren't quite the same as pure software emulation like qemu (though you could potentially use qemu for this too), but VM software does implement this snapshot system, so clearly it's possible.

You know when the bug is triggered through the visual output. And what's cool is that by saving periodic snapshots automatically to a ring buffer, you can code a special keypress to "rewind" the program. So it crashes, then you go back a bit, and save a disk snapshot there, and wait to see if the bug repeats. If it does, then you turn on trace logging and dump all of the CPU instructions between your save point and the crash. Then you go to the crash point, and slowly work your way back to try and find out where things went wrong.

Byuu is the author of the excellent SNES (Super Nintendo) emulator bsnes. I would guess he is referring to emulation errors in emulators, where snapshot support is common, and usually of good quality.

Well, nothing, but that's bug hunting: you try it and see. Whether or not this helps to recreate the problem on demand, you know more than you did before.

Reversible debugging might help more here than snapshots, since it allows you to record a failure and then debug it: http://rr-project.org/

Probably nothing, but it's worth a try.

Nothing, but it reduces the conditions of the variables, which is handier than nothing.