News for hackers. Stuff that matters.
Re:Cerebras Systems unveils a 1.2T transistor chip for AI by modeless on Monday August 19, 2019 @01:43PM (#20739905)
There are far more transistors in this chip than neurons in the human brain. In 100 of these chips, there are more transistors than there are synapses in the human brain.

I don't mean to suggest that transistors are equivalent to neurons or synapses; clearly they are very different things (though it is not clear that neurons are much more computationally powerful than transistors, despite assertions from many people that this is the case). But I think it is still useful to compare the complexity of structure of chips vs. the human brain. We are finally approaching the same order of magnitude of complexity in structure.

Also note that this is not manufactured on TSMC's highest density process. Assuming TSMC's 3nm process development is successful, that will probably be 6x denser than the 16nm process used here.

Re:Cerebras Systems unveils a 1.2T transistor chip for AI by jacquesm on Monday August 19, 2019 @03:18PM (#20740875)
A few buckets of sand also contain more elements than there are synapses in the human brain. They clearly lack any structure, and so are computationally stuck at a big fat '0' no matter what you do with the sand (unless you want to turn it into a giant abacus or turn it into integrated circuits).

Where on the scale between a few buckets of sand and an actual working human brain this chip is is not so much a function of it's structure but of what it does, and the brain's finer structures are so complex that even modelling a few neurons is going to crush that complexity downwards by so many orders of magnitude that we simply hope that we are not accidentally throwing out the useful bits in the simulation.

That's a roundabout way of saying that I think that:

> We are finally approaching the same order of magnitude of complexity in structure.

Is not necessarily true in a way that is relevant. The parts count or basic interconnection may have nothing to do with how the brain is connected internally, nor with how it functions at the lowest levels.

Re:Cerebras Systems unveils a 1.2T transistor chip for AI by Veedrac on Monday August 19, 2019 @05:17PM (#20741916)
> A few buckets of sand also contain more elements than there are synapses in the human brain.

Actually you'd need about a thousand buckets.

https://www.quora.com/How-many-grains-of-sand-could-fit-in-a...

---

Ultimately I think these numbers do matter. As chips approach the size of human brains, only 1,000,000,000x faster, you remove the question of engineering and scale, and are left with just research.

Re:Cerebras Systems unveils a 1.2T transistor chip for AI by azinman2 on Monday August 19, 2019 @05:34PM (#20742030)
We don’t know what numbers are the numbers that we need to care about. How much do chemical gradients matter? Inhibitors? Hormones? What other factors are at play?

I read so many people that have both simplified views of DNNs as well as real neurons that they get overly excited about the possibility of somehow getting to AGI by just having enough transistors. There was similar thinking in the 1950s when proto-AIs could play chess.

Cognition is extraordinary. The theory of mind encompasses many important features that we have no use how to approach.

To say all that’s left is research doesn’t really say much at all. It’s like saying for teleportation all that’s left is research as we have lots of bandwidth, or that for month-long battery life all that’s left is research as we have the phones ready to take them.

The gap between our knowledge Of the brain and what’s being applied to machine learning is a big leap.

Re:Cerebras Systems unveils a 1.2T transistor chip for AI by Veedrac on Monday August 19, 2019 @05:49PM (#20742145)
Brains run on computation. Computation is fungible. The highest bandwidth signal in the brain is through neuron spiking, and there isn't much space for other mechanisms to hide. Take a simple 10 virtual neurons:1 synapse ratio as a rough upper bound of how much computation the brain can be doing, and by necessity the rest is architecture.

This is not at all like teleportation. It is absurd to claim numbers much larger than my own. 100:1? Where would the computation be? It is completely unnecessary to claim that the brain must have vast amounts of hidden capacity to do what it does, rather than the secret being in the ‘software’, and it goes against most of what we know about the brain, its ancestry, and computation.

This is not like teleportation. The brain isn't magic. I don't have to claim to know how the brain works to point this out.

Re:Cerebras Systems unveils a 1.2T transistor chip for AI by ypcx on Tuesday August 20, 2019 @09:43AM (#20746912)
For the in-brain computation, this may be true. However there may be non-local quantum effects emergent from the functioning of this massively parallel biological neural network apparatus, and information could be received and transmitted outside of the five senses. Whether this information or functionality is necessary for basic functioning of the brain, we don't know yet. I'm guessing it is, but I'm also guessing that we can supplant it by little more computation.
Re:Cerebras Systems unveils a 1.2T transistor chip for AI by AstralStorm on Tuesday August 20, 2019 @01:30PM (#20749574)
And that butterfly just caused a tornado that killed the original poster.

Quantum effects, while nonlocal, are small magnitude, not even microtubules are sensitive enough for this at the temperatures brain runs at.

It is nigh certain that mind does not depend on this magnitude of effects. There's way too much chaos inside even a single neuron, much less a whole network.

Re:Cerebras Systems unveils a 1.2T transistor chip for AI by colordrops on Tuesday August 20, 2019 @12:07AM (#20743754)
Is anything magic? It doesn't really matter when a thing is so inscrutable that it is indistinguishable from magic, especially if you believe that your personal subjective experience is 100% the result of brain function.
Re:Cerebras Systems unveils a 1.2T transistor chip for AI by Veedrac on Tuesday August 20, 2019 @08:33AM (#20746175)
Maybe it's best to wait until after we've hammered away at the problem on warehouses of human-scale hardware for 50 years before calling it inscrutable? The paper that put neural networks on GPUs is ten years old. I get that the problem looks hard, but so do lots of things.
Re:Cerebras Systems unveils a 1.2T transistor chip for AI by ohum on Monday August 19, 2019 @07:01PM (#20742539)
Please don’t take this as a personal attack. Your comment and certainty sound like religious fanaticism. The belief that consciousness is merely computation is unproven, and a matter of faith. It always seems a little short sighted when people claim to know something about consciousness and the mind based on a belief that the brain amounts to a biological computer. As an example of an alternative possibility to brain as a biological computer: in a holographic universe the brain itself is an emergent system. Or, Buddhists say that the material world as we perceive it is interdependent with mind, which is more fundamental than form. Some interesting phenomena that present problems with mind as programming running on a biological computer are the placebo/*cebo effects and Wim Hof :). Mind over matter, man.
Re:Cerebras Systems unveils a 1.2T transistor chip for AI by The_rationalist on Monday August 19, 2019 @08:45PM (#20742970)
qualias (consciousness, feelings, senses) might be impossible to create through transistors. But it does not matter, we can build something functionally equivalent in a computable way. This is called: a philosophical zombie https://en.m.wikipedia.org/wiki/Philosophical_zombie
Re:Cerebras Systems unveils a 1.2T transistor chip for AI by ohum on Monday August 19, 2019 @11:39PM (#20743638)
A computing machine (not taking environmental factors like cosmic rays into account (though, why not?)) is deterministic, unless it interfaces with a truly random source of entropy. My physicist friend has said, there may be no randomness, only complexity.

Is all deterministic or do conscious entities have free will, or...?

The decisions made by your philosophical zombie will either be predetermined by the programming, or seeded by randomness. The decisions made by such a machine cannot be proved to be functionally equivalent to the decisions made by a conscious entity, and belief in such is a matter of faith.

Belief in randomness itself is a matter of faith. It’s impossible to prove that any event or measurement is truly random, and yes, there may be hidden variables- there’s no way to test.

There is no rational reason to believe that conscious behavior, consciousness, feelings, and all the activity of the mind, are functionally equivalent to a program and/or randomness.

Sounds like you are believer in the religion of materialism.

Re:Cerebras Systems unveils a 1.2T transistor chip for AI by lostmsu on Tuesday August 20, 2019 @02:26AM (#20744294)
> There is no rational reason to believe that conscious behavior, consciousness, feelings, and all the activity of the mind, are functionally equivalent to a program and/or randomness.

On the contrary, the number of tasks, that a human can do, but a machine can't is shrinking. That recently started including art.

Re:Cerebras Systems unveils a 1.2T transistor chip for AI by The_rationalist on Tuesday August 20, 2019 @08:40AM (#20746242)
Is all deterministic or do conscious entities have free will? Free will vs determinism is a fun debate for beginners in philosophy. But it is not a debate per se, the answer is obvious once asked seriously in a well defined manner. So firstly you will agree with all scientists on earth that the world, that matter is causal and totally predictable on each of it's properties though calculus. https://en.m.wikipedia.org/wiki/Causality_(physics) Something contra-causal has never been observed in all humanity history. The conception that a human (made of matter after all) is totally deterministic is consistent and explain human behavior. It is epistemologically weak to ask an alternative (free will) that needs another premise. Let's define free will : "Some conceive free will to be the capacity to make choices in which the outcome has not been determined by past events." This means: the free choice come not from the past, it comes from nowhere. To make such a thing possible a brain would need to create primary cause (sorry the English Wikipedia page does not yet exists https://fr.m.wikipedia.org/wiki/Cause_premi%C3%A8re). A prymary cause is a cause that come from nowhere and create consequences and causes that are determined by the primary cause. Only the primary cause has "free will power". So are there primary cause in the universe? Happily not, otherwise science would be broken and the world chaotic, unpredictable. There is one true candidate for primary cause: the big bang. To think that a brain is able to break the law of physics to create a primary cause is maybe not impossible but highly unlikely, and totally unscientific, mostly religious Many things can be said this premise: When does it start? Does a bacteria has this power? An ant? A fish? A cat? Or only humans? Because language allow free will? Or because language allow Suffisant complexity to hidden the deep causes between or choices and create an illusion of choice? So free will has no explanatory power, determinism is the most observed thing empirically, only humans have it? (because it is trivial to model deterministically mammals) which add even more ad-hoc-ness. Another question is: what is the frequency of free will on a human ? Are 80% of your thoughts free? Or PI%? Another ad-hoc Ness but the more frequency of free will you believe the more epistemologically weak your belief is.

But even after all this, let's say that free will exist, what does it change? Can it allow things that determinism cannot? Firstly free will is NOT free.. Imagine you ask me to solve a problem. Let's say that knowledge about A is necessary to find a solution. If I don't know A and is undeducible from my prior knowledge then I can't solve the problem, free will or not. My past is unsufficient, and a primary cause cannot give me knowledge (sadly ;)) Humans choices and thoughts are bounded by knowledge. They are also bounded by fluid intelligence. Humans have two single goals in their lifes: Maximize their happiness and maximize others Hapinness. It is the only meaningful thing in life (qualias). So it is important to understand that for each situation in life there is an optimal choice and maybe competing <= choices. The goal of a human is to find the most optimal choice, it necesssiate a lot of knowledge and a lot of rationality training e.g learn cognitive biases, logical fallacies, skepticism, the scientific method, learn more words, etc. The more you gain erudition and rationality the less choices appear as interesting as what you know is the most optimal you can find. The ideal omniscient person has no choice to make, she knows what choice is optimal. A choice is a choice when you don't know enough the consequences and so you make a bet (which e.g can maximize risk/return or minimize it) Once you know the optimal choice what does free will means? It means the freedom to make an irrational, worse choice than the optimal one. Such a useless concept (when you know the optimal one)

Free will would only apply on a list of choices when you cannot rank any of those choices, you don't know if any choice from the list is better than any other. In such a (rare) situation, free will would allow true randomness of choice. What's the point? And more than that, it's refuted. Humanity has built pseudo random number generators that are by far good enough and if you wanted from such a list of choice make a random choice, you better use a PRG! Because if we empirically measured your believed random choices, they would not at all be random. A common proof of that is the game paper scissor rock where people have big difficulties to not repeat patterns.

So I have shown that 1) free will is one of the weakest possible belief on earth, epistemologically. And 2) that it has no explanatory power (does not explain even one thing that determinism cannot) And 3) that it necessarily reduce to true randomness and that it would useless (cf "what's the point") and that it is empirically shown to be wrong. Sadly, when you have since a long time a strong belief, with emotional affect, reading a sound refutation does not allow the reader to change it's belief. You have probably not the cognitive freedom to now thinks "free will doesn't exist and if it would it would be useless" (see the irony about freedom?)[1] because human brains are buggy. Firstly when not trained we have an inability to see (on others and on ourself) logical fallacies. https://en.m.wikipedia.org/wiki/List_of_fallacies

And we cannot see our cognitive biases https://en.m.wikipedia.org/wiki/List_of_cognitive_biases which is in itself a bias called the blind spot bias.

[1]This does not apply or less, if you are trained to rationality. Lesswrong.com Rationalwiki.org And Wikipedia are good places

"Sounds like you are believer in the religion of materialism." You got me! Well to be precise, materialism is now called physicalism and it is synonymous to be a believer of the scientific method. I'll take that as a compliment. But it is anthitetic to" religion", Popper would have a heart attack reading your sentence ^^ You know what, you are a physicalist too even if you ignore it. All human progress is driven by science, it would be time to recognize that.

Re:Cerebras Systems unveils a 1.2T transistor chip for AI by jacquesm on Monday August 19, 2019 @05:25PM (#20741974)
Using fine sand I make that out to be about 6 buckets, give or take, besides, this isn't about whether it is 6, 10 or even 100 (or 1000) buckets, it is about how the number itself does not make a huge difference. Before if you wanted 1.2T transistors you needed 50 or so regular large dies, and presumably there have already been dies with more than the 21 billion that I'm referring to here (the NVidia 5K cores chip).

If that was all it took then we'd have had 'human brains' in a box long ago, after all, 50 GPUs is something that isn't all that rare in plenty of institutions.

Re:Cerebras Systems unveils a 1.2T transistor chip for AI by Veedrac on Monday August 19, 2019 @05:34PM (#20742031)
There are 100 trillion synapses in the brain.

I'm not claiming Cerebras is human scale; I think it's a factor of 100-1000 off personally. I'm just saying I don't think the comparison is meaningless.

Re:Cerebras Systems unveils a 1.2T transistor chip for AI by jacquesm on Monday August 19, 2019 @05:37PM (#20742057)
Time will tell. I personally put the time between now and AGI well in the decades, possibly more than a few centuries. This is not a mere matter of engineering, it is one of fundamental understanding of which we likely have achieved only very little.

Imagine seeing the wiring diagram of an alien computer with parts on the order of 6 orders of magnitude smaller than the ones that you are familiar with without knowing the contents of its memory and then you are expected to make a working copy. Good luck.

Re:Cerebras Systems unveils a 1.2T transistor chip for AI by neural_thing on Monday August 19, 2019 @03:39PM (#20741063)
I have written a book about the true computational power of pyramidal neurons. Spoiler: they are A LOT more powerful than transistors. The book is free @ http://www.corticalcircuitry.com/
Re:Cerebras Systems unveils a 1.2T transistor chip for AI by thelittleone on Monday August 19, 2019 @05:07PM (#20741838)
I'm a few pages in and with found it fascinating, humorous, compelling and understandable even with zero background in the field. Thanks for sharing.
Re:Cerebras Systems unveils a 1.2T transistor chip for AI by neural_thing on Monday August 19, 2019 @05:45PM (#20742118)
Thank you for the kind words!
Re:Cerebras Systems unveils a 1.2T transistor chip for AI by christoph on Tuesday August 20, 2019 @08:32AM (#20746159)
I made the mistake of starting to read this in bed last night and got totally engrossed. Looking forward to finishing it off over the next few days. Many thanks for writing this and sharing it.
Re:Cerebras Systems unveils a 1.2T transistor chip for AI by plainOldText on Tuesday August 20, 2019 @02:19AM (#20744264)
I have started reading the first two chapters of your book and I really like it. It's well written and easy to understand. If it continues to be this wonderful for the remaining chapters I will buy it as well. Thanks for writing it.
Re:Cerebras Systems unveils a 1.2T transistor chip for AI by jdsully on Monday August 19, 2019 @02:45PM (#20740527)
I haven’t seen a perceptron circuit done in less than 4 transistors plus a few diodes. However these chips are almost certainly digital and will use a lot more to perform the floating point math.

Single transistors can’t even compute our simplified model of a neuron let alone the complexities of the real thing.

Re:Cerebras Systems unveils a 1.2T transistor chip for AI by mlyle on Monday August 19, 2019 @03:03PM (#20740709)
If we're going to talk about net computational power per unit (neuron/transistor), we also need to note that single transistors do stuff a whole lot faster than biological neurons, too. If we had neurons that were 1,000x faster, could we use fewer of them for the same task?
Re:Cerebras Systems unveils a 1.2T transistor chip for AI by jacquesm on Monday August 19, 2019 @03:19PM (#20740891)
Well, we can turn that argument around: if an artificial brain could be faster than a real one, does that mean that right now we have slow artificial brains that are just as good as the real ones? The answer being an obvious 'no' may help answer your question.
Re:Cerebras Systems unveils a 1.2T transistor chip for AI by mlyle on Monday August 19, 2019 @03:45PM (#20741127)
Just because we don't know a program that creates a decent artificial brain doesn't mean that such a program doesn't exist for the hardware we have.

We're almost certainly going to be able to make speed/size trade-offs in AGI, because we're used to making them in all other computing.

Even biological systems make these types of trade-offs. We're beings optimized for the environment we evolved in, not optimized for "smarts". There's many "choices" in energy balance, reaction time, etc, that have been made.

Transistors are really powerful and fast computing devices; really our use of them in digital logic in some ways greatly under-utilizes them for ease in design and predictability. Saying that it takes 4 transistors to make a really simple neuron analog in a circuit undersells that those 4 transistors can operate many, many orders of magnitude faster.

Re:Cerebras Systems unveils a 1.2T transistor chip for AI by jcims on Monday August 19, 2019 @05:36PM (#20742048)
Faster than what? There are a hundred trillion atoms in a neuron operating in a coordinated fashion to create its global behavior in the context of the brain. How many interactions between those components occur in a billionth of a second? Many.
Re:Cerebras Systems unveils a 1.2T transistor chip for AI by tlb on Monday August 19, 2019 @04:13PM (#20741369)
It would be very hard to tell if an AGI running 1000x slower than a brain were intelligent. Humans babies don’t talk for about a year, but nobody has tried firing up a putative slow AI and waiting for 1000 years.
Re:Cerebras Systems unveils a 1.2T transistor chip for AI by jacquesm on Monday August 19, 2019 @04:24PM (#20741450)
Then let's run 1000 of them in parallel, right?

You're on to the exact point I'm trying to make. The whole idea that 'just' complexity or 'just' speed is enough to make something intelligent that is not yet intelligent is off-base to a degree that resorting to metaphors to try to explain why also breaks down.

I liken intelligence to life: once it exists it is obvious and self-perpetrating but before it exists it is non-obvious and no amount of fantasizing about what it would take to make it will get you incrementally closer until one day you've got it. But that day will start out just like the thousands of days before it. It's a quantum leap, a difference in quality, not an incremental one. Yes, enough order of magnitude change can make a quantitative change into a qualitative one. But so far the proof that this holds for intelligence is eluding us. Maybe we're just not intelligent enough, in the same way that you can't 100% describe a system from within the system.

Re:Cerebras Systems unveils a 1.2T transistor chip for AI by tlb on Monday August 19, 2019 @04:47PM (#20741663)
But it doesn't parallelize. Watching 1000 babies for their first 0.365 days isn't going to reveal their potential.

More practically, deep learning experiments were tried in the 80s, but the results weren't encouraging. If they'd left the experiments running for a sufficiently long time, they might have gotten great results. But they gave it a few days, saw the loss rate plateau and hit ^C, and that was that.

Most likely, the early deep learning experiments had some parameters wrong. Wisdom about choosing parameters only came once people could run multiple experiments in a week. So it's likely the same with some hypothetical AGI. Early experiments will have some parameters wrong, and it'll either go into seizures or catatonia. We won't be able to get the parameters right until we can run experiments faster than human brains. Say, simulating 5 years of life in a week, or 250x real time.

Re:Cerebras Systems unveils a 1.2T transistor chip for AI by jacobush on Monday August 19, 2019 @05:41PM (#20742088)
Wow, that is a thought I didn't have before. Thanks!

Still, we wouldn't even know what real time is, or what the "x" is, rather, until after we have achieved AGI. Also a mind bender.

Re:Cerebras Systems unveils a 1.2T transistor chip for AI by tlb on Monday August 19, 2019 @06:21PM (#20742344)
Right, we have no idea how much computation it takes. Discoveries usually start with a less efficient algorithm, and then the performance gradually improves once we know what it needs to do. We might need 1000x the compute power to discover something as it'll eventually need once we deploy it. So I'm a big fan of ridiculously fast hardware like this article describes.
Re:Cerebras Systems unveils a 1.2T transistor chip for AI by jacquesm on Monday August 19, 2019 @04:51PM (#20741702)
Yes, we are indeed in violent agreement.
Re:Cerebras Systems unveils a 1.2T transistor chip for AI by mlyle on Monday August 19, 2019 @04:45PM (#20741641)
If it's possible to implement AGI on a computer, a 4004 with enough RAM attached can do it... very slowly. So the speed/size trade-off is implicit unless we take a pseudo-spiritualist view that computing machines just can't be intelligent.

There's two big options out there: either it requires much, much more computation than we can readily employ even to do it relatively "slowly", or we don't know how.

The former has a little bearing on the balance between a transistor's computational capabilities and a neuron's, but even so is largely orthogonal.

As to parallelism, it doesn't work that way. If you get nine women pregnant, you don't get a baby per month. If we've made a working but far-too-slow-for-us-to-realize-it AGI, making more of them doesn't help us understand the problem.

Re:Cerebras Systems unveils a 1.2T transistor chip for AI by jacquesm on Monday August 19, 2019 @04:51PM (#20741712)
> we don't know how

Exactly.

Re:Cerebras Systems unveils a 1.2T transistor chip for AI by crististm on Tuesday August 20, 2019 @04:42AM (#20744890)
I think that part of the idea of adding more speed and data to solve the AI problem started with Norvig's presentations on how more data resolved the search problem better than using more clever algorithms.

People are using the 'more data' approach because it makes a (minor) dent at the problem. It's the only tool they have right now.

It is my opinion that it is not enough to make us smarter in understanding the quantum leap necessary.

Re:Cerebras Systems unveils a 1.2T transistor chip for AI by trott on Monday August 19, 2019 @04:07PM (#20741322)
> If we had neurons that were 1,000x faster, could we use fewer of them for the same task?

Absolutely. In fact, we do: in batched processing (pervasive in deep learning today), CNNs and other weight sharing schemes (e.g. transformers) the same "neurons" get re-used many times.

Re:Cerebras Systems unveils a 1.2T transistor chip for AI by mkl on Monday August 19, 2019 @08:37PM (#20742940)
Those aren't neurons, though. Biology is different, synaptic connections have to physically fit, etc. Individual neurons serve multiple purposes, but I don't think we know if their speed is a limiting factor in that.
Re:Cerebras Systems unveils a 1.2T transistor chip for AI by berdon on Monday August 19, 2019 @04:05PM (#20741305)
You're mixing terminology.

A neuron is not a synapse.

- There are ~100 billion neurons in the human brain

- There are 100-1000 trillion synapses in the human brain

There's strong evidence to suggest that each synapse is more akin to a digital perceptron than a neuron is. Synaptic cleft distance, transmitter types, dendritic structure, reuptake, etc are all factors that can allow for some level of long-term storage and mediation of subsequent neural activation.

[Edit] Maybe not "mixing" but seemingly comparing apples and oranges.

Re:Cerebras Systems unveils a 1.2T transistor chip for AI by Simon321 on Tuesday August 20, 2019 @01:51AM (#20744167)
It takes around 15-20 transistors to make a simple neuron simulation:

https://www.quora.com/How-many-transistors-can-be-used-to-re...

But: "Synapses are usually separately modeled in transistors (they are not part of the neuron circuits described above) and dramatically add to the transistor count."

"[A neuron] has on average 7000 synaptic connections to other neurons."

So we're not there yet, but it's definitely an improvement!