You could well be right. In my own tests, the ESP swap is what tanked performance. A more limited stackless test did not have nearly the same overhead. And indeed, the largest speedup was from moving to set ESP as soon as possible, get the return address into EAX, and finally jump to it. CALL+RET with late ESP set behaved poorly, leading me to believe it was due to the CPU stalling on the ESP change.
I did also try inlining this, but it was much less portable and actually performed slower in my emulator. I won't claim that will be the case for everyone, of course. YMMV.
Wow. I used my own stack switch routine for several years now and didn't realize it could be improved until I read this thread. Thank you very much.
This is my old naive version:
https://github.com/ademakov/MainMemory/blob/master/src/base/...
This is what I have now:
https://github.com/ademakov/MainMemory/blob/cstack-switch-re...
You are missing x/y/zmm regs in the clobber list.
Yes, that's true. However in my app this has never caused any problems. Apparently even if the compiler auto-vectorizes anything it does this only with scratch regs or in leaf procedures. I don't have any fp or simd code of myself.
A big penalty is due to (failed) branch prediction: a coroutine switch is essentially an indirect branch so the predictor might need help (for example using a different branch instruction address for each coroutine type); also call instruction used to call into the coroutine switch function is not paired with a ret instruction, which messes with the specialized call predictor. Changing the final jmp instruction in the switch function to push add; ret actually makes thing worse; the best solution is to inline the switch function in the caller via inline assembler (this also helps with the previous issue and, if the compiler provides the functionality, it allows only saving the registers that are actually in use)
edit: last time I did a synthetic benchmark of my one of my own coroutine implementations, the switch performance was only constrained by the number of taken branches the CPU could issue (one every other cycle): https://github.com/gpderetta/delimited/blob/master/benchmark...
[1] I'm talking about the typical x86 cpu.