> do we have any ideas what the limits are on the algorithm side of things?

I've always believed that humans do not have the ability to program things smarter than themself, because we do not understand our own intelligence, so we have no way to reproduce it.

At the time, I said the only alternative I can think of is make random permutations and pick the best one, and go from there. But I said this as a ridiculous suggestion that no one would actually do.

But then Google actually did exactly that with their Go machine!

So, that's what I think: The future of computing will be based on randomness and the job of the programmer will be to guide it, but not program it directly. (Can you imagine programming a webpage this way? Or writing a book this way?)

(Totally self-promoting here) I did a lot of work along this line in my PhD -- stochastic architectures for probabilstic computation. http://ericjonas.com/pages/circuits.html There's an increasing amount of interest in this space.

You're work is interesting. The concepts remind me of work in analog implementations of neural circuitry. See, the brain is likely a general-purpose, analog computer with digital-like parts. So, trying analog implementations was thought to have improvements. It did with one wafer-scale method I didn't see coming.

http://yann.lecun.com/exdb/publis/pdf/boser-92.pdf

http://www.kip.uni-heidelberg.de/Veroeffentlichungen/downloa...

It would be interesting to see someone combine the principles of your work with analog implementations on a decent process node. Yours is kind of like a hybrid between properties of analog and digital cells. The real thing might be even more effective albeit harder to automate. There's some analog EDA but it's almost always custom work.

Sounds like really neat work. It reminds me of an idea I've toyed with just a little: you know how chemical reaction networks can compute? E.g. https://johncarlosbaez.wordpress.com/2014/03/23/programming-... Last I heard, they'd used DNA strand displacement in vitro for simple neural network stuff. So I wondered how practical it'd be to encode a Bayes net as a reaction network, with the Gibbs sampling done directly by thermodynamics. This wouldn't compete with the sort of thing we do in silicon; maybe it'd work out for tiny diagnostic systems in synthetic bio? With less overhead than the neural nets.

(Well, that took more space than I wanted. I downloaded one of your circuits papers for a gander.)

The point of machines is that they do a single task better than us. A dishwasher is superhuman at washing plates but nothing else. If we model machines to be more like us they will end up with the same disadvantages of a normal human. At that point why not employ real humans in the first place?

> At that point why not employ real humans in the first place?

Cost mostly, If you can replicate a human-level intelligence for 10,000 (or 100,000) and have it work 24/7 with no time off and scale out to thousands of them then you'd have something absolutely terrifying in it's capability.

Image an AWS of a 1000 von-neumann level intelligences working co-operatively 24/7 on a problem.

The stuff of sci-fi right now but maybe one day, we know it's physically possible to build human level intelligences since we prove that the rest is 'just' engineering.

While we may not have the ability to program things 'smarter' than themselves, we do have the ability to create tools that extend our intelligence. Machine learning, randomness, and even new javascript frameworks all fall under this umbrella.

>But then Google actually did exactly that with their Go machine!

I would argue that this is not what they did. What they did was take something a human player does (study games to learn how to play). And then parallelized this to a level that a human can't match. Basically the equivalent of having one go player playing against an entire team of go players who are all experts.

Humans every day "program" things smart than themselves. Every parent who raises a child smarter than themselves does exactly that. Smarter than human level computers will be created the same way we create smart humans.

I do not think this is equal. Babies may actually be "brighter" than parents, without the parents having to actually "do" anything to produce the higher IQ baby. But building an artificial system means some humans actually have to design it.

Yes parents have to do something to end up with brighter children than themselves - if you put a child in a dark room from birth with no human interaction I can guarentee that they will not come out very bright 20 years later (there have been some very sad examples of this in history).

The point I was making is that intelligent agents are built all the time by less intelligent agents. Humans will build more intellgent AI using exactly the same approach and these AI's can do the same.

Of couse building an AI via evolutionary training opens up a whole lot of control issues. Do we really want to be creating entities vastly more intelligent than ourselves when we have no real idea of their interests and when we occupy something that is valuble (e.g. matter and energy) to this entity?

It is worth remembering that just because we are reaching some physical limits of "classical" chip design. For example, both x86 and arm date back to the 80s. While I have no doubt that the implementation of these architectures has improved to reflect modern manufacturing capabilities, this still suggests that there is room for architectural improvements in performance.

Beyond simple architectural improvements, we could still move beyond basic transistor based computing. The most common example is quantum computing (which offers an asymptotic improvement in some cases), however I can imagine there beyond other classical devices that can compute certain functions more efficiently than a pure transistor based solution can.

> For example, both x86 and arm date back to the 80s.

Modern x86 and high performance ARM cores are almost unrecognizable compared to processors in the late 90s, much less the 80s. (Also, ARM was founded in 1990, not the 80s).

> While I have no doubt that the implementation of these architectures has improved to reflect modern manufacturing capabilities, this still suggests that there is room for architectural improvements in performance.

There are still performance improvements to be had, but it's not going to be anywhere near the performance scaling of Moore's law. The rate of architectural improvements is also slowing down as well (and increasingly only applicable for a smaller and smaller fraction of workloads).

> I can imagine there beyond other classical devices that can compute certain functions more efficiently than a pure transistor based solution can.

Like...? Quantum is 10+ years away right now and unlikely to get fast any time soon. CMOS has had decades and billions of dollars invested in scaling; it's going to be a long time before any of the current "CMOS killers" (virtually all of which are still transistors) reach parity.

From wikipedia on ARM:

The official Acorn RISC Machine project started in October 1983.

But everything else you say is bang on.

> Beyond simple architectural improvements, we could still move beyond basic transistor based computing. The most common example is quantum computing (which offers an asymptotic improvement in some cases)

The number of problems for which quantum computing offers a speedup is very limited. It is absolutely not a general, all-purpose computational architecture.

They are working on optic chips that use photons. https://www.sciencedaily.com/releases/2015/08/150813142710.h...

Having X86 and ARM chips means you can run legacy code on them from a long time ago. Which is why they are so popular. But once we go to optical chips for quantum computing there will be no legacy apps from X86 or ARM on them. It would have to write apps from scratch.

Windows 10 is the last version of Windows for a reason. Microsoft is porting their enterprise stuff like SQL Server to Linux. https://blogs.microsoft.com/blog/2016/03/07/announcing-sql-s...

You can tell that Microsoft knows that Windows is getting long in the tooth, and has to support legacy code, and still has old code in it for compatibility reasons. They look at other operating systems like Linux to port their enterprise apps to in order to sell tech support and take a stab at Oracle, MySQL, and PostgreSQL. They know that Linux gets ported to different platforms and so can SQL Server to give them a larger marketshare.

The old X86 and ARM designs are limited due to legacy support of older programs. But they are marketed as backward compatible with older chips.

IBM's Power has been open sourced as OpenPower. http://openpowerfoundation.org/

The Dragonball CPU tries to build on the M68K family of processors. https://en.wikipedia.org/wiki/Freescale_DragonBall

So you got a lot of backward compatible CPUs out there, that are limited because they have to support legacy code. The new processors that don't have backward compatibility should run faster with fewer quirks and use new technology not from the 1980s, but programs have to be written from scratch or ported from other platforms.

Windows 10 is the last version of Windows because Microsoft knows that it will eventually have to drop compatibility in order to compete with the new systems and new processors out there that don't have legacy support. The X86-64 chips are a dead-end, and Microsoft has to look to other technologies and a different operating system. Linux is a good choice to support even if Microsoft does not officially have a Linux distro yet. If they open source their enterprise tools like Dotnet or CLR or Roslyn or Visual Studio Code SQL Server to Linux and OSX and other platforms they can sell tech support for it via their paid hotlines. Even making iOS and Android apps for Office and other things.

Microsoft is going to move away from X86-64 and Windows eventually, and focus on The Cloud instead and Azure in hosting VPS operating systems. Then when the new design of processors come out that put X86 and ARM to shame they can port their programs to that new platform.

If you remember the original Micro-Soft business model was to make programs for computers that other companies made and make them for different operating systems. They only got into DOS because IBM made them an offer they couldn't refuse. Windows was basically their attempt at making a Mac GUI for DOS, and working with IBM to bring OS/2 was yet another GUI attempt, but they quit OS/2 and focused on Windows instead. OS/2 was going to be ported to PowerPC, MIPS, Alpha, SH4 and other RISC processors because IBM and Microsoft saw the limitations of the 80X86 processors and wanted something new. But it fell apart. Then Windows NT 4.0 was ported to MIPS, Alpha, etc but abandoned. Windows RT was ported to ARM but flopped. Every attempt to move away from 80X86 processors met with disaster because people wanted to run legacy code.

But soon it will be a new day with new processors and new computers not based on 1980s designs and using Linux or some other FOSS OS and connecting with Cloud computers.

A ton of it did. There's many subtopics for algorithms, microarchitecture tricks, hardware accelerators, I/O schemes, improvements for RTL/transistor optimization, and so on. Each have enough papers it can be hard to fibd stuff. Most of best stuff gets patented and controlled by dominant companies.