As solid as your argument may be, everyone saw arguments like this over and over. Every single time they were solid. For a time, it was the high frequency noise that would not be manageable (80s), then heat dissipation (90s), then limits on pipeline optimization (00s) and now size constraints on transistors. They were all hard barriers, deemed impossible and all were overcome.

I already know that your answer will be: "but this time it is a fundamental physics limit". Whatever. I'm jaded by previous doomsday predictions. We'll go clockless, or 3D, or tri-state or quantum. It'll be something that is fringe, treated as idiotic by current standards and an obvious choice in hindsight.

This looks like a good example of the Normalcy bias logical fallacy: https://en.wikipedia.org/wiki/Normalcy_bias

That previous constraints have been beaten in no way supports the argument that we will beat the laws of physics this time.

Our brains use roughly ~20 watts though, so we know that the power constraints can be overcome, if not in silicon then it may be biological machines we use in the future.

The previous problems were solved because people were willing to spend hundreds of billions of dollars to solve them. And they are still spending that kinds of money.

If the normalcy bias was in effect, they wouldn't be spending that money.

Actually Normalcy Bias may in fact feed that kind of money spending until such time as reality hits. Assuming that people will automatically act more logically when large amounts of money is in play flies in the face of recent history. Just look at the recent housing loan crisis. Normalcy Bias played a part there.

It's certainly possible that we'll break more barriers with clever engineering and new scientific breakthroughs. But that doesn't mean the Normalcy Bias isn't in play here.

Normalcy bias may have people spending lots of money on fabs assuming that the problems would be solved by the time the fabs are built.

However, I'm talking hundreds of billions spent on R&D to specifically to solve problems associated with chip manufacture. It took on the order of 25 years to solve each of the problems listed in the grandparent's post. Nobody would spend that kind of money or time on something that they think somebody else would solve.

People have probably spent billions of dollars to find a cure for cancer, but there isn't one that works for all cancers and most are still very bad news.

Say you spent a hundred billion dollars to extinguish the sun- that wouldn't work. How much money you spend is irrelevant when you're up against what people call "hard physical limits".

Isn't our inability to cure all cancers a limitation of our knowledge more than a hard physical limit?

I've read several articles saying that different cancers are not exactly the same disease, but more like different diseases with the same symptom (uncontrolled tumor growth) and different etiology, even sometimes different from person to person, not just from tissue to tissue. This was said to be a reason that a general cancer cure is so elusive. But is it really thought of as impossible, not just elusive?

Maybe our inability to extinguish the sun is also a limitation of knowledge more than a hard physical limit!

Even if I'm right about this, your description of the situation would still be accurate in that there would be no way to simply throw more money at the problems and guarantee a solution; there would need to be qualitative breakthroughs which aren't guaranteed to happen at any particular level of expenditure. If people had spent multiples of the entire world GDP on a space program in the 1500s, they would still not have been able to get people to the moon, though not because it's physically impossible to do so in an absolute sense.

>> there would need to be qualitative breakthroughs which aren't guaranteed to happen at any particular level of expenditure

Yep, that's my point, thanks. Sorry, I'm not in my most eloquent today :)

And the cost of building a fab is increasing exponentially; eventually that trend has to come to an end.

It also looks like a fully general argument against anything new ever being accomplished.

There is a lot of room for improvement with the implementation. The way we are using deep neural networks at the moment is exellent for prototyping, but far from optimal. For instance, this paper http://arxiv.org/abs/1511.00363 shows that you can replace floating point operations by simple bitwise operations without losing too much precision in DNNs for image recognition. Together with a better (that is, compiled, instead of interpreted) representation of the inference step I would expect an order of magnitude improvement at a small loss in precision. More software tuning, especially the kind of low-level optimizations that most chess programs do, should yield another big improvement.

Finally, the hardware we are using to run these programs is insane. Sure the silicon is approaching some hard physical limits, but your processor spends most of that power trying to make old programs run fast...

My prediction is that with enough ressources it is possible to write a Go AI which runs on general purpose hardware that's manufactured on current process nodes and fits in your pocket.

I don't think you appreciate how much of this is good algorithms, and how little you need sheer computing power to get good results.

If you look at http://googleresearch.blogspot.com/2016/01/alphago-mastering... you'll find that Google's estimate of the strength difference between the full distributed system and their trained system on a single PC is around 4 professional dan. Let's suppose that squeezing it from a PC to a phone takes about the same off. Now a pocket phone is about 8 professional dan weaker than the full distributed system.

If their full trained system is now 9 dan, that means that they can likely squeeze it into a phone and get a 1 dan professional. So the computing power on a phone already allows us to play at the professional level!

You can get to an unbeatable device on a phone in 10 years, if self-training over a decade can create about as much improvement they have done in the last 6 months, AND phones in 10 years are about as capable as a PC is today. Those two trade off, so a bigger algorithmic improvement gets you there with a weaker device.

You consider this result "overly optimistic". I consider this estimate very conservative. If Google continues to train it, I wouldn't be surprised if there is a PC program in a year that can beat any Go player in the world.

You're right, it won't be a general purpose computing device the way we conceive of it with the von Neumon architecture.

It'll likely be hardware that can be generalized to run any kind of deep net. The iPhone 5S is already capable of running some deep nets.

As a friend mentioned, it isn't the running of the net, it's the training that takes a lot more computational power (leaving aside data normalization). A handheld device that is not only capable of running a deep net, but also training one -- yeah, that will be the day.

There are non von Neuman architectures that are capable of this. Someone had figured out how to build general-purpose CPUs on silicon made for memory. You can shrink down a full rack of computers down into a single mother board, and use less wattage while you are at it.

This really isn't about having a phone be able to beat a Go player. Go is a transformative game that, when learned, it teaches the player how to think strategically. There is value for a human to learn Go, but this is no longer about being able to be the best player in the absolute sense. Go will undergo the same transformation that martial arts in China and Japan has gone through with the proliferation and use of guns in warfare.

Rather, what we're really talking about is a shot at having AIs do things that we never thought they could do -- handle ambiguity. What I think we will see is -- not the replacement of blue collar workers by robots -- but the replacement of white collar workers by deep nets. Coupled with the problems in the US educational system (optimizing towards passing tests rather than critical thinking, handling ambiguity, and making decisions in face of uncertainty), we're on a verge of some very interesting times.

Your making the same assumption people made about computing in the 50s, then 70s, then 90s, etc.

Please do elaborate. I try to base my assumptions (which I accept may turn out to be completely wrong) on physics and experience in working in semiconductors.

I just don't see a 1000x+ decrease in the power required happening in a decade or two without some revolutionary technology I can't even imagine. Is this what you meant? I'm sure most people couldn't imagine modern silicon chips in the 1950s vacuum tube era. But now we're getting close to the theoretical, well-understood minimums in silicon chips, so another revolutionary step is required if another giant leap like that is to be achieved.

    > physics and experience in working in semiconductors

    > without some revolutionary technology I can't even
    > imagine

I suspect (in the nicest possible way) that in a lineup of your imagination (on current assumptions) vs the combined ingenuiety of the human race driven by the hidden hand, the latter wins.

> > Give it ten years and […]

> I find this overly optimistic

exDM69 never said it's not gonna happen, he just said that it's not going to happen in ten years, and I agree with him. Revolutions never occurs that quickly. To achieve that we don't just need an improvement of the current state of the art, we need a massive change and we don't even know what it's going to look like yet ! This kind of revolution may occur one day but not in ten year.

And it could even never happen, remember that we don't have flying cars yet ;)

The thing is though we could already be 10+ years along the path to that next revolution, it wont start being talked about until its basically here

It seems to me that the people who say "it won't happen" do tend to have a much better reason to say it won't happen (or rather that it _probably_ won't happen) than the people who insist the next big revolution is just round the corner just because the last big revolution did happen.

The optimistic position is a bit like saying: "I 've lived 113 years, I'm not going to die now!". It's entirely possible for a trend to reverse itself. If machine learning has taught us something is that background knowledge (in this case, of processor technology) gives you much better results than just guessing based on what happened in the past.

Here's some possibilities:

Stacked 3D chips (HBM, etc), Heterogenous computing (OpenCL, Vulkan), Optical computing, Memristors, Graphene-based microchips, Superconductors, Spintronics, Quantum computers, Genetic computers (self-reconfigurable)

Heterogenous computing is already used in AlphaGo (and your smartphone). 3d chips will come to mainstream devices in a few years, but will give "only" a modest performance boost, say 2x or so.

The rest of the technologies you mention have great potential but will they be available in a smartphone in one decade? I don't think so.

You might ultimately only need some specialized "neural processing instruction set" for either the GPU cores or for the CPU cores. Or at least, I don't see any obvious obstacles to that.

I feel the same way about the chips reaching their physical limits. But I keep waiting for a new way we use them. We used to just churn out MHz and that was the metric. Then we got hyper-threading, multi-cores, GPU and other specific processors and new ways of programming to go with it all. I imagine we'll see the same. Just like the brain has different areas of processing, I'm hoping we'll see the same in silicon chips. Just like how we offload work from the general purpose cpu to the more efficient purpose build gpu or sound card etcs. Not saying every computer is going to have a GO chip in it, but maybe someday we'll have machine learning processors or who knows what. But yeah the advancements will be new designs and new ways of processing instead of more power.

Sure. But so far, we've found that revolutionary step every time we've hit these sorts of walls, and if I was a betting man I'd wager we'll do the same again.

Right, but just as a contrast: Technological progress speed has been at an all-time high since the begin of the industrial revolution.

It might as well slow down again and we have to remember that most humans in history saw little to no advances in technology over their lifetime.

I'm excited for the possibilities modern science opens up but I also think we might reach a point where fundamental progress stalls for a century or two.

I guess free worldwide information transfer (aka Internet) just opened this era and we are not close to see any kind of stalling (IMHO).

Amongst other things, you're assuming hardware is where the speed will come from. But it's as likely to come from better software.

How many watts does Lee Sedol's brain require?

About 25.

(2000 kilocalories / day -> ~100W; the brain uses about a quarter of your calories.)

A Go app likely wouldn't rely on the native processing power of the smartphone. An AlphaGo app could be created today for a smartphone. The bottleneck isn't the phone it's the cost of the cloud computing resources behind it. Perhaps a combination of Moore's law and economy of scale would make it affordable sooner than we think. The Xbox One, for example, already subcontracts difficult problems out to Azure.

The unbeatable GO app on your phone doesn't have to do the processing locally.

Yes, but that's just a silly argument and definitely not what GP meant. You can go and play a Go bot on KGS network with your smartphone today.