In other words, could I write an epiphany webserver?
I get that GPUs are a bad fit for many problems, but how fast can this thing be if it has no local cache? Will this really be faster than a high-powered Xeon, or will it just consume less energy per operation? Is it a teaching tool? A tech demo? Something that makes Erlang/Haskell magically run much faster?
EDIT: And if you want to read/write the same memory areas from multiple cores, it's your own responsibility to either get the timing right, or use other means (e.g. you can trigger interrupts in another core) to signal when it is safe for another core to access data.
You can address core-local memory, memory in another core, or main memory the same way - the difference is speed.
There's no cache, and you're responsible for avoiding race conditions in memory access yourself.
The main CPU is substantially faster than the Pi, but it doesn't have HW accelerated graphics, so it's not a speed daemon for desktop/workstation type use.
As for the Epiphany, assume that it'll be slower than most GPU's for tasks that GPU's are good for. That is, if you can make do with few instruction streams, the Epiphany is not well suited for it, as most GPUs will blow the current chips out of the water in terms of performance.
If, on the other hand, your problem is poorly suited for GPUs due to lots of independent instruction streams, it may be better suited.
One of the most interesting aspects of the Epiphany is that is can also be connected into a grid - each chip has four high speed links that can be connected to other Epiphany chips, or be used to interface with the main CPU or off-chip memory.
The cores can all access each-others memory without any special instructions, including that of the cores on other Epiphany chips that are hooked up via the external links - the only difference between in-core and out-of-core memory access is the speed.