What "longest wait" measures is (b_i - a_i) in:
Time a_0
lock
Time b_0
unlock
Time a_1
lock
Time b_1
unlock
...
lock
Time a_0
unlock
lock
Time b_0
Time a_1
unlock
lock
Time b_1
...
The fact that the timings are supposedly different on a millisecond scale is therefore quite suspicious.
Perhaps the chosen clock ends up calling a syscall, which can end up in the scheduler on syscall return occasionally and throw off the measurements that way? Or just the fact that there's one extra syscall in the critical section sometimes causes other threads to make different yielding decisions which throws off the scheduling?
More generally, it would have been great to see a deeper investigation into scheduling behavior, which is possible on Linux using the tracing infrastructure and visualizing with different tools like Hotspot.
Getting a worst-case delay of ~1.5ms with a naive ticket_spinlock that can yield is really not surprising, to be honest. All it takes is for patterns like:
1. 3 threads pull their tickets in order. 2. The middle thread yields and something else is chosen to run by the scheduler, or perhaps the scheduler decides not to run anything for a short while on the off-chance that something else becomes runnable, which is a fair choice because after all, you yielded. 3. The last thread is then blocked by the middle thread.
It just boils down to the fact that "yield" is a bad primitive. You should fall back to futex-based synchronization instead so that the kernel has a better idea of what you're trying to do. The article comes to the same conclusion anyway, in the form of "just use std::mutex".
My guess is that if the thread gets descheduled inside the lock() spin loop (either by an explicit call to yield() or otherwise) then the "Four longest idle times" reflect the fact that all of the worker threads have remained descheduled for an extended period. But assuming that the threads have elevated priority (why would you use spinlocks on non-high-priority threads?) this seems implausible on an 8 core machine.
I would be curious to know what the results are if isolcpus was used and the workers were explicitly assigned to those CPUs.