In this particular case you should be able to use no-op instructions to spin without consuming much power. Since the consumer only spins when the queue is empty we know there will be space on the ring buffer and producers won't be impacted. When the consumer wakes from a no-op loop and finds work on the queue it can switch back to a hot loop for a certain amount of time before returning to a no-op cold loop.

My understanding is that even having very small no-op pauses can significantly reduce the amount of energy used, and crucially heat generated, while only having a very modest impact on latency. A good fit for bursty low latency systems.

To be clear, I am only relating something that another developer described to me. I've not actually implemented this myself. :)

I think your colleague is talking about the PAUSE instruction (which is also known as REP NOP, since they encode to the same bytes). It's a special instruction that hints to the processor that it's in a spin-loop waiting on a synchronisation variable to change. It's used in tight wait loops like this:

    wait_loop: 
        pause
        cmp eax, sync_var
        jne wait_loop

The PAUSE instruction introduces a small delay to synchronise the spin loop to the memory bus frequency. This is a power savings, since the value of sync_var, as seen from the perspective of this code, can't change faster than that. Also, because the CPU can execute much faster than the memory bus, it prevents many memory requests from piling up while in the loop (because of out-of-order execution; those requests will have to be unrolled when the loop exits), making the loop faster to exit. Because of Hyperthreading, the pause also gives another thread a brief opportunity to execute on the same core.

https://software.intel.com/sites/default/files/m/d/4/1/d/8/1...

Intel does, however, recommend other approaches to dealing with tight loops like this. See here:

https://software.intel.com/en-us/articles/long-duration-spin...

How would there be many requests? Wouldn't it load the cache line once into the shared state and then spin waiting for the line to be invalidated before reloading?

Honestly, I don't know how this interacts with cache lines.

As far as I know, Intel has not released any official details about what the PAUSE instruction does other than that it slows down spin loops to a reasonable rate. The best source I know of for this information is the Intel® 64 and IA-32 Architectures Optimization Reference Manual (http://www.intel.com/content/dam/www/public/us/en/documents/...). It is fairly vague about what the instruction does, but gives some useful information in 13.5.3 "Spin-Wait Loops", as well as 8.4 "Thread Synchronization", and in particular 8.4.2 "Synchronization for Short Periods" which says:

"On a modern microprocessor with a superscalar speculative execution engine, [a spin loop] results in the issue of multiple simultaneous read requests from the spinning thread. These requests usually execute out-of-order with each read request being allocated a buffer resource. On detection of a write by a worker thread to a load that is in progress, the processor must guarantee no violations of memory order occur. The necessity of maintaining the order of outstanding memory operations inevitably costs the processor a severe penalty that impacts all threads."

So it seems that the concern here is that (without a PAUSE) the speculative execution engine effectively unrolls the spin loop and executes a sequence of reads on the sequence variable. This queues up a list of pending memory operations that need to be unrolled in order to ensure that the result is the same as if they were executed sequentially -- even though in this case it wouldn't make any difference.

I think the harm here is for other threads on the same core that share the same queue of in-flight loads.

Coherence wise it won't interfere with other cores and their access to memory.

That said how hyper-threads share CPU resources is a moving target. My understanding through hearsay is that in the past many resources were statically partitioned with hyper-threading enabled, but now things are moving towards allocating resources dynamically which means that a thread wasting resources would be sucking up capacity that could be used by another hyper-thread.

I was unaware of the PAUSE function. It certainly did belong inside my spin-loop. The other recommendation of cache-line alignment I was following, and it does make a difference.

I take my hat off to you a-priori

Glad to help :)

My loop continually checked the buffers and the locks, it never did a noop. Those fenced instructions are probably not the most power efficient. Power consumption was not a concern, latency was our primary goal.

It is very unfortunate. On that project we looked at all sorts of ways to avoid burning cores but they all had significant latency issues. If it involved going to the kernel then there was a major context switch overhead. But the big killer is the non-constant time. Each time you go to the kernel there's a danger of loosing your timeslice or being shunted to some other core. This leads to very high variance in performance. (I must always repeat though, this variance is not likely relevant in most applications.)

So we spun in loops. And we had the CPUs set to maximum performance (no idle-time scaling). The total real load was probably less than 1%, so yes, it's very unfortunate.

It would seem like MONITOR/MWAIT (https://blogs.oracle.com/dave/resource/mwait-blog-final.txt) would be the best of both worlds: extremely low-power and low-latency. But I see very little about it. Are there reasons that it's not more widely used?