In the mid 1980s there was a CPU called a "Transputer" [1] made some of the people who moved to ARM. These CPUs could be connected together in huge networks and directly talk to each other.

The network of CPU's could auto-discover its topology, but coding for so many CPU's was difficult. Some specific algorithms scaled well with the number of CPUs, but most did not.

[1] https://en.wikipedia.org/wiki/Transputer

I think the reason transputers didn't 'make it' is not because they were super hard to program (it was only a little bit harder than programming a regular computer), but because the price premium you paid for a transputer set-up was too high and x86 got faster very rapidly.

This is right around the time when the first 386 machines were launched and in a very short time we went from 12-20 MHz 286 boxes (and some 68K machines for the lucky ones) to 33 MHz 386 machines with a ton of RAM.

So the advantage that transputers had was eroded very quickly and I don't think INMOS was ready to match pace.

Now that we've reached the end of the line for that kind of speed increase we are seeing a renewed interest in multi-cpu fabric architectures, of which the transputer was an instance.

You could replace "transputers" with [insert-architecture] and wind up with a nice overarching theory to explain the history of computer hardware.

Is ARM ( or something else) going to surpass x86 (amd64 whatever you want to call it) in the near future?

And that these will affect all ISA equally.

Transputers never really scaled into this sort of league, they were more 8/16/32 type scenarios..

For the interested, there was also the Atari Transputer Workstation (ATW).

...Although interesting, they were expensive and didn't perform compared to the equivalent sequential machine. Given development tho, who knows where it could have gone.

Also, the auto-discovery wasn't really auto... That was the boot code probing for CPUs on other ends of links and propagating itself to connected CPUs, building a map of the network in the process (which is kind of cool).

[1]http://store.steampowered.com/app/370360/

Looked great though: http://www.corestore.org/cm2a.htm

The Connection Machine Lisp programming language described in Daniel Hillis' PhD dissertation was essentially going to be Lisp with parallel map/reduce but AFAIK was never done being implemented.

I think the big problem was that most of the SIMD algorithms were yet to be discovered at the time. For example this paper by Hillis and Steele was a very big deal but looks kind of basic today: http://uenics.evansville.edu/~mr56/ece757/DataParallelAlgori...

Guy Blelloch did a lot of work on Connection Machines and basically wrote the book on SIMD programming: http://www.cs.cmu.edu/~blelloch/papers/Ble90.pdf He also made a very nice programming language for parallel computing (NESL), as did Gary Sabot who worked at Connection Machines (Paralations). When you compare those to Hadoop or OpenCL it really is a wonder where we went wrong and what the designers of the latter were thinking (or not).

So it's more of a data flow model than a function-call model, and every node will transform its input rather than that it will return results to callers.

This means that not every problem is easily mapped to such an architecture, but when there is a good correspondence between the topology of the network of CPUs and the problem then the throughput can be very good.

After all, there is no bottleneck where all of these have to access a shared resource (RAM) so when it pays off it pays off big.

The developers of the PETSc system wrote a position paper eschewing the typical hybrid (MPI+threading) techniques for trundling towards "exascale" -- if with MPI's shared memory support, which doesn't require cache coherence. (Apart from remote memory access, MPI departs from actor-/CSP-like systems in supporting things like collective communication (and i/o), and dynamic process management.)

There are 12 memory modules on the chip, so I assume each is shared by less than 100 cores, which is close to a beefy multithreaded server chip. Each core also has a small local program memory that's not shared between them. It'd be not unlike programming a networked cluster of processing nodes each with a separate chunk of memory, except that the cores could be assigned to neighboring workloads.

Sounds fun!

  - http://texasmulticore.com/

And, yes, I know: Not open source. No one needs to make that clever observation...