[1] See the Laws of Orders paper.
[2] without using asymmetric synchronization which is not appropriate for a general purpose barrier.
1. Seq-cst -- Strongest/slowest. (Aka: Java volatile)
2. Total Store order (aka: x86)
3. Acq-Release (aka: newer-ARM / newer-POWER)
4. Acq-Consume (aka: old-ARM / old-Power)
5. Relaxed -- Weakest/fastest, DEC Alpha.
That is to say, acquire-release is often compiled into no-op on x86, because x86 Total-store ordering is "stronger" than acquire-release.
Acquire-Consume was thought to be sufficient, but C++11 debate argued otherwise, and designed a stronger Acquire-release model. ARM/POWER designers then implemented acquire-release model.
--------
I understand that things aren't perfectly linear, and that at the lower level the discussion is about read/read, read/write, write/read, and write/write barriers. But my brain has never really grok'd this level of detail. (Instead, I think in terms of the simplified list above). So please correct me if I'm wrong with the above assumptions.
ARM and power do allow for indipendent reads of indipendent writes (or maybe only power) so in principle an acq+rel fence could be cheaper than a full seqctst one, although I don't know if it is in practice.
This summarises (https://community.arm.com/arm-community-blogs/b/architecture...):
"The Armv8.0 support for release consistency is based around the “RCsc” (Release Consistency sequentially consistent) model described by Adve & Gharacholoo in [1], where the Acquire/Release instructions follow a sequentially consistent order with respect to each other. This is well aligned to the requirements of the C++11/C11 memory_order_seq_cst, which is the default ordering of atomics in C++11/C11."
"Instructions are added as part of Armv8.3-A to support the weaker RCpc (Release Consistent processor consistent) model where it is permissible that a Store-Release followed by a Load-Acquire to a different address can be re-ordered. This model is supported by the use of memory_order_release/ memory_order_acquire /memory_order_acqrel in C++11/C11."
https://aloiskraus.wordpress.com/2018/06/16/why-skylakex-cpu...
1. PAUSE should only be executed inside of the spinlock.
2. Fast-path won't touch the "PAUSE" instruction. (Ex: lock is unlocked. So you grab the lock, bypass the while loop and carry on).
3. Slow-path is... well... the slow-path. You'll be spinning in there for thousands of cycles, waiting for "the other thread". Saving power by effectively "sleeping" a bit longer isn't a big issue.
-----
I very well could assume that Intel did the measurements, and that PAUSE with 100+ delay of latency was higher performance / throughput than their original PAUSE implementation. It is a "slow-path" instruction after all.
If the spin-lock has any contention what-so-ever, you're looking at 100+ cycles to even communicate to L3 cache (~40-nanoseconds == 160 cycles to talk to L3 cache at all).
It makes no sense to be "any faster" than this, because L3 cache hasn't changed yet. You might as well sleep until the next point-in-time where L3 cache had a chance to update (aka: the value of the spinlock has changed).
L2 cache and L1 cache are "private" to each core, so its impossible for another core to change the values in L1 or L2. So the "closest" the spinlock's value can be is L3 cache. The only exception I can think of... is if both threads are SMT/Hyperthreads on the same core. This case has probably gotten slower, but all other cases have probably gotten faster (or more precisely: higher throughput).
A lot of the "difficult parts" of the barrier is the synchronization / memory-barrier part (not the same as a spin-lock barrier. Ugggh, we need someone to revamp the language of low-level multiprocessor programming).
Anyway, the memory-barrier part is "solved" by using atomics and then handwaving all the loads/stores as sequentially-consistent. Yeah, that works, but would lead to subpar performance on all systems, as x86 has total-store ordering (less powerful than sequentially consistent), so x86-compilers would probably need to emit a memory barrier.
ARM / POWER8/POWER9/POWER10 systems are "even weaker", so they need an even more powerful barrier to be emitted by the compiler.
-------
There's also the "Hyperthread yield" instruction that should be placed somehwere in POWER/x86 systems. (Most ARM systems largely don't have hyperthreads/SMT, but the ARM E2 does...). Without "hyperthread yield", the pipeline is going to be uselessly full of load/store instructions from this tight inner-loop.
EDIT: I had to search for it, but I now know the name of the instruction on x86. "PAUSE" is the hyperthread-yield instruction. https://www.felixcloutier.com/x86/pause
You may be "livelock starving" your brother-hyperthread because your current thread is using up all of the pipeline / decoding bandwidth. By using "hyperthread yield", your code will use the minimum amount of decoding bandwidth.
-----
The article does talk about memory-barriers, but seems to be missing the discussion on hyperthread yield. So its probably not "as efficient" as it could get yet.
I can believe that the implementation works as discussed in the blogpost. But its still yet incomplete: a faster implementation probably exists, maybe not with "Fully Relaxed" atomics, but probably with acquire/release atomics (which are free on x86, and significantly faster on ARM/POWER9 than seq-cst atomics).
Note: ARM8+ and POWER9+ now have load-acquire and store-release instructions, serving as an intermediate performance between fully relaxed atomics and seq-cst atomics. (These instructions may have existed earlier, but I know for sure they're in the modern version of these chips).
Once acquire/release barriers are figured out in the right location, the use of hyperthread-yield on x86 and POWER systems would be best on highly-loaded systems. I'm not sure how to test the efficacy / throughput of hyperthread-yield however, I only know of its theoretical benefits. Coming up with a good test (to make sure we're actually improving performance) is itself a difficult question.
TL;DR: Good first pass implementation, and good discussion to get things started. But there's still a long way before I consider it a top-end implementation! For many projects, what is seen here is probably sufficient (especially professional projects where you need to move onto some other programming task). But a hobby project could keep optimizing a bit longer to squeeze out a few more cycles of efficiency here.
Coming up with a test to ensure the acquire-release is properly done is also somewhat difficult, and would require ARM / POWER hardware.