It comes back to the programming model. Synchronization is all explicit. See publication list. Includes work on MPI, BSP, OpenMP, OpenCL, and OpenSHMEM. The work from US army research labs on OpenSHMEM is especially promising. It's a PGAS model.

OpenSHMEM paper: https://arxiv.org/abs/1608.03545

got it, thanks. it looks like the per-node memory controller has an atomic test and set

edit: and also a global wired-or for a barrier.

If you're looking for weird synchronization primitives, look at the documentation of the DMA controller. It has a mode in which it stores bytes that are written to a particular address in a memory range in order the writes arrive. I haven't figured out a reasonable way to use that with multiple writers (except the trivial case of having a byte-based stream with bounded size), though.

Yeah, I was thinking about that problem too. (It's not safe to blindly write somewhere unless you can be sure that nobody else is going to simultaneously clobber your data. You can't do any kind of atomic test-and-set or compare-and-swap operation on remote memory, so you don't have the usual building blocks for things like queues or semaphores.)

The problem becomes a lot easier if you can reduce the multiple-writer case to the single-writer case. One idea that occurred to me is that since you have 1024 cores, it might make sense to dedicate a small fraction of them (say, 1/64) to synchronization. When you need to send a message to another process, you write to a nearby "router" that has a dedicated buffer to receive your data. The router can then serialize the with respect to other messages and put it into the receiver's buffer.

Basically, you'd end up defining an "overlay network" on top of the native hardware support; you pay a latency cost, but you gain a lot of flexibility.

EDIT: I may be completely wrong about the first paragraph; it looks like the TESTSET instruction might actually be usable on remote addresses. I assumed it didn't because the architecture documentation doesn't say anything about how such a capability would be implemented. But if it works, it would drastically simplify inter-node communication.

IIRC TESTSET is usable: IIRC it just sends a message that causes that to happen, but you don't learn if the test succeeded.

I was talking about the DMA mode in which every write to special register (that may be coming from a different core) gets "redirected" to subsequent byte of the DMA target region. This can work as a queue with multiple enqueuers, but has bounded size (after the size is exhausted, messages get lost) and operates on single byte messages.

The easiest way to think about it is that remote access is order-preserving message passing with a separate message network for reads (as it truly is), so: 0. Local reads and writes happen immediately. 1. Writes from core X to core Y are committed in the same order in which they happen. 2. Reads of core Y from core X are performed in the same order in which they are executed, and they are performed sometime between when they get executed and their result is used. 3. Reads can be reordered WRT writes between the same pair of cores (so you _don't_ see your writes).

I don't remember how does this work with external memory (including cores from different chips).

Not a hardware genius here. What does coherent memory mean?

As the other comments have said, it basically has to do with the level of consistency between different processors' views of the shared memory space. (There are some semantic differences between "consistency" and "coherence" that I'm going to ignore.)

For some context, the x86 memory model gives you an almost consistent view of memory. The behavior is roughly as if the memory itself executes reads/writes in sequential order, but writes may be buffered within a processor in FIFO order before being actually sent to memory. Internally, the memory actually isn't that simple -- there are multiple levels of cache, and so forth -- but the hardware hides those details from you. Once a write operation becomes globally visible, you're guaranteed that all of its predecessors are too.

From what I can see from a quick overview of the Epiphany documentation, it doesn't have any caches to worry about, but it gives you much weaker guarantees about memory belonging to different cores. For one thing, there's no "read-your-writes" consistency; if you write to another core and then immediately try to read the same address, you might read the old value while the write is still in progress. For another, there's no coherence between operations on different cores, so if you write to cores X and then Y, someone else might observe the write to Y first (e.g. because it happens to be fewer hops away).

It applies to architectures with caches https://en.wikipedia.org/wiki/Cache_coherence

Epiphany-V does not have caching. You explicitly move data around in software. Some software abstractions are better than others.

As I understand it: If memory is coherent then all cores see the same values when they read the same location at the same time. Stated another way, the result of a write to a location by one core is available in the next instant to all other cores, or they block waiting for the new value.

Thank you all for that help. Did not see definitions elsewhere in post.