If you want to control the computer, you have options. Qemu can give you fine grained CPU level logging you want: http://moyix.blogspot.jp/2014/07/breaking-spotify-drm-with-p... -- this is an article that walks through logging them instructions. (https://github.com/panda-re/panda)
Finally there are RE frameworks where you can instrument (hook) function calls (Frida being a good cross platform option).
I would try AFL first. But playing with Panda can be really fun too.
(edit: and a colleague just pointed out -- https://github.com/angr, which will also let you work right at the level you want to, I think).
A hit tracer sets a breakpoint at the beginning of every basic block, and records and clears every breakpoint hit (so the performance hit is relatively low).
You'd use a hit tracer to record a "baseline" of your target program when it's not doing the thing you care about. Then you'd run the program again and trigger the behavior. Then you'd diff the traces.
AFL is doing something related when it explores states in targets.
There are a lot of free hit tracers, but they're also kind of a build-your-own-light-saber deal. If you have a Python debugger library and a disassembly, a hit tracer is close to "hello world".
You could write this with Pin [1], but I'd be surprised if there wasn't a profiling tool with instruction level analysis available. If there truly isn't, then there are Pin examples that can be pretty quickly modified to achieve this.
1. https://software.intel.com/en-us/articles/pin-a-dynamic-bina...
There was one thing in particular where I knew there was a jump somewhere (if some_length < some_width) that caused bad outputs. I was playing around looking at registers etc in gdb while following along with a disassembled version of the code, but it was impossible to get any idea where to start.
I wanted something that could give me a few seconds worth of samples of where the instruction register was spending its time as a starting point, but couldn't find any such tool (linux).
Within my control:
Does anyone have any advice on how you might approach such a situation?