Right. The point of the API is to trigger rendering as late as possible before the next frame swap, not immmediarely after the last frame swap, as the author expected.
To do so, it has to estimate how long your app’s callback will take to render a frame. (E.g. if it expects your callback to render a frame in 10ms it might call it 11ms before the next frame swap.)
It seems the heuristic it uses to estimate the duration of the callback execution is assuming a fairly constant duration. The documentation seemed to suggest that. But as the author has inconsistent rendering duration.
So, e.g., if it estimates 10ms and therefore calls back 11ms before the frame swap but you end up taking 20ms, you’re going to drop a frame. If it then adjusts and calls back 21ms before, but you render in 1ms, you could render two frames in the space of one displayed frame.
Anyway, just thinking through it aloud.
I think the author may not want to use this API for his purpose, though perhaps by adjusting his rendering so that it always takes the “max” rendering time might let its estimates be correct and then things would smooth out.
I'm also not actually working on anything that uses this right now, I was just curious. I'm just warning about the dangers if you have inconsistent rendering time.
I think the bigger concern is latency between HID input and visual feedback, which people are more sensitive to, and which you can’t schedule in advance for.
http://powerpcliberation.blogspot.com/2013/04/os-x-disable-y...
It's not that it calls you as late as possible though, as my experiments show it just calls you at an arbitrary but consistent point in the frame like a timer.
It also says ”In the past, synchronizing your video frames with the display’s refresh rate was often a problem, especially if you also had audio. You could only make simple guesses for when to output a frame […] which didn’t take into account possible latency from user interactions, CPU loading, window compositing and so on. The Core Video display link can make intelligent estimates for when a frame needs to be output, based on display type and latencies. […] If for some reason the processing takes longer than expected (that is, the display link’s estimate is off), the video graphics card can still drop frames or otherwise compensate for the timing error as necessary.”.
So, disassembling the code wasn’t strictly necessary.
Combining this with this article, it seems they try to call your program as late as possible, possibly to decrease latency between the time you have influence on what is displayed and the time it is displayed. Could that especially if you also had audio be the reason they do that, as it might make it easier to keep audio in sync with video?