Where on the scale between a few buckets of sand and an actual working human brain this chip is is not so much a function of it's structure but of what it does, and the brain's finer structures are so complex that even modelling a few neurons is going to crush that complexity downwards by so many orders of magnitude that we simply hope that we are not accidentally throwing out the useful bits in the simulation.
That's a roundabout way of saying that I think that:
> We are finally approaching the same order of magnitude of complexity in structure.
Is not necessarily true in a way that is relevant. The parts count or basic interconnection may have nothing to do with how the brain is connected internally, nor with how it functions at the lowest levels.
Actually you'd need about a thousand buckets.
https://www.quora.com/How-many-grains-of-sand-could-fit-in-a...
---
Ultimately I think these numbers do matter. As chips approach the size of human brains, only 1,000,000,000x faster, you remove the question of engineering and scale, and are left with just research.
I read so many people that have both simplified views of DNNs as well as real neurons that they get overly excited about the possibility of somehow getting to AGI by just having enough transistors. There was similar thinking in the 1950s when proto-AIs could play chess.
Cognition is extraordinary. The theory of mind encompasses many important features that we have no use how to approach.
To say all that’s left is research doesn’t really say much at all. It’s like saying for teleportation all that’s left is research as we have lots of bandwidth, or that for month-long battery life all that’s left is research as we have the phones ready to take them.
The gap between our knowledge Of the brain and what’s being applied to machine learning is a big leap.
This is not at all like teleportation. It is absurd to claim numbers much larger than my own. 100:1? Where would the computation be? It is completely unnecessary to claim that the brain must have vast amounts of hidden capacity to do what it does, rather than the secret being in the ‘software’, and it goes against most of what we know about the brain, its ancestry, and computation.
This is not like teleportation. The brain isn't magic. I don't have to claim to know how the brain works to point this out.
Quantum effects, while nonlocal, are small magnitude, not even microtubules are sensitive enough for this at the temperatures brain runs at.
It is nigh certain that mind does not depend on this magnitude of effects. There's way too much chaos inside even a single neuron, much less a whole network.
Is all deterministic or do conscious entities have free will, or...?
The decisions made by your philosophical zombie will either be predetermined by the programming, or seeded by randomness. The decisions made by such a machine cannot be proved to be functionally equivalent to the decisions made by a conscious entity, and belief in such is a matter of faith.
Belief in randomness itself is a matter of faith. It’s impossible to prove that any event or measurement is truly random, and yes, there may be hidden variables- there’s no way to test.
There is no rational reason to believe that conscious behavior, consciousness, feelings, and all the activity of the mind, are functionally equivalent to a program and/or randomness.
Sounds like you are believer in the religion of materialism.
On the contrary, the number of tasks, that a human can do, but a machine can't is shrinking. That recently started including art.
But even after all this, let's say that free will exist, what does it change? Can it allow things that determinism cannot? Firstly free will is NOT free.. Imagine you ask me to solve a problem. Let's say that knowledge about A is necessary to find a solution. If I don't know A and is undeducible from my prior knowledge then I can't solve the problem, free will or not. My past is unsufficient, and a primary cause cannot give me knowledge (sadly ;)) Humans choices and thoughts are bounded by knowledge. They are also bounded by fluid intelligence. Humans have two single goals in their lifes: Maximize their happiness and maximize others Hapinness. It is the only meaningful thing in life (qualias). So it is important to understand that for each situation in life there is an optimal choice and maybe competing <= choices. The goal of a human is to find the most optimal choice, it necesssiate a lot of knowledge and a lot of rationality training e.g learn cognitive biases, logical fallacies, skepticism, the scientific method, learn more words, etc. The more you gain erudition and rationality the less choices appear as interesting as what you know is the most optimal you can find. The ideal omniscient person has no choice to make, she knows what choice is optimal. A choice is a choice when you don't know enough the consequences and so you make a bet (which e.g can maximize risk/return or minimize it) Once you know the optimal choice what does free will means? It means the freedom to make an irrational, worse choice than the optimal one. Such a useless concept (when you know the optimal one)
Free will would only apply on a list of choices when you cannot rank any of those choices, you don't know if any choice from the list is better than any other. In such a (rare) situation, free will would allow true randomness of choice. What's the point? And more than that, it's refuted. Humanity has built pseudo random number generators that are by far good enough and if you wanted from such a list of choice make a random choice, you better use a PRG! Because if we empirically measured your believed random choices, they would not at all be random. A common proof of that is the game paper scissor rock where people have big difficulties to not repeat patterns.
So I have shown that 1) free will is one of the weakest possible belief on earth, epistemologically. And 2) that it has no explanatory power (does not explain even one thing that determinism cannot) And 3) that it necessarily reduce to true randomness and that it would useless (cf "what's the point") and that it is empirically shown to be wrong. Sadly, when you have since a long time a strong belief, with emotional affect, reading a sound refutation does not allow the reader to change it's belief. You have probably not the cognitive freedom to now thinks "free will doesn't exist and if it would it would be useless" (see the irony about freedom?)[1] because human brains are buggy. Firstly when not trained we have an inability to see (on others and on ourself) logical fallacies. https://en.m.wikipedia.org/wiki/List_of_fallacies
And we cannot see our cognitive biases https://en.m.wikipedia.org/wiki/List_of_cognitive_biases which is in itself a bias called the blind spot bias.
[1]This does not apply or less, if you are trained to rationality. Lesswrong.com Rationalwiki.org And Wikipedia are good places
"Sounds like you are believer in the religion of materialism." You got me! Well to be precise, materialism is now called physicalism and it is synonymous to be a believer of the scientific method. I'll take that as a compliment. But it is anthitetic to" religion", Popper would have a heart attack reading your sentence ^^ You know what, you are a physicalist too even if you ignore it. All human progress is driven by science, it would be time to recognize that.
If that was all it took then we'd have had 'human brains' in a box long ago, after all, 50 GPUs is something that isn't all that rare in plenty of institutions.
I'm not claiming Cerebras is human scale; I think it's a factor of 100-1000 off personally. I'm just saying I don't think the comparison is meaningless.
Imagine seeing the wiring diagram of an alien computer with parts on the order of 6 orders of magnitude smaller than the ones that you are familiar with without knowing the contents of its memory and then you are expected to make a working copy. Good luck.
Single transistors can’t even compute our simplified model of a neuron let alone the complexities of the real thing.
We're almost certainly going to be able to make speed/size trade-offs in AGI, because we're used to making them in all other computing.
Even biological systems make these types of trade-offs. We're beings optimized for the environment we evolved in, not optimized for "smarts". There's many "choices" in energy balance, reaction time, etc, that have been made.
Transistors are really powerful and fast computing devices; really our use of them in digital logic in some ways greatly under-utilizes them for ease in design and predictability. Saying that it takes 4 transistors to make a really simple neuron analog in a circuit undersells that those 4 transistors can operate many, many orders of magnitude faster.
You're on to the exact point I'm trying to make. The whole idea that 'just' complexity or 'just' speed is enough to make something intelligent that is not yet intelligent is off-base to a degree that resorting to metaphors to try to explain why also breaks down.
I liken intelligence to life: once it exists it is obvious and self-perpetrating but before it exists it is non-obvious and no amount of fantasizing about what it would take to make it will get you incrementally closer until one day you've got it. But that day will start out just like the thousands of days before it. It's a quantum leap, a difference in quality, not an incremental one. Yes, enough order of magnitude change can make a quantitative change into a qualitative one. But so far the proof that this holds for intelligence is eluding us. Maybe we're just not intelligent enough, in the same way that you can't 100% describe a system from within the system.
More practically, deep learning experiments were tried in the 80s, but the results weren't encouraging. If they'd left the experiments running for a sufficiently long time, they might have gotten great results. But they gave it a few days, saw the loss rate plateau and hit ^C, and that was that.
Most likely, the early deep learning experiments had some parameters wrong. Wisdom about choosing parameters only came once people could run multiple experiments in a week. So it's likely the same with some hypothetical AGI. Early experiments will have some parameters wrong, and it'll either go into seizures or catatonia. We won't be able to get the parameters right until we can run experiments faster than human brains. Say, simulating 5 years of life in a week, or 250x real time.
Still, we wouldn't even know what real time is, or what the "x" is, rather, until after we have achieved AGI. Also a mind bender.
There's two big options out there: either it requires much, much more computation than we can readily employ even to do it relatively "slowly", or we don't know how.
The former has a little bearing on the balance between a transistor's computational capabilities and a neuron's, but even so is largely orthogonal.
As to parallelism, it doesn't work that way. If you get nine women pregnant, you don't get a baby per month. If we've made a working but far-too-slow-for-us-to-realize-it AGI, making more of them doesn't help us understand the problem.
Exactly.
People are using the 'more data' approach because it makes a (minor) dent at the problem. It's the only tool they have right now.
It is my opinion that it is not enough to make us smarter in understanding the quantum leap necessary.
Absolutely. In fact, we do: in batched processing (pervasive in deep learning today), CNNs and other weight sharing schemes (e.g. transformers) the same "neurons" get re-used many times.
A neuron is not a synapse.
- There are ~100 billion neurons in the human brain
- There are 100-1000 trillion synapses in the human brain
There's strong evidence to suggest that each synapse is more akin to a digital perceptron than a neuron is. Synaptic cleft distance, transmitter types, dendritic structure, reuptake, etc are all factors that can allow for some level of long-term storage and mediation of subsequent neural activation.
[Edit] Maybe not "mixing" but seemingly comparing apples and oranges.
https://www.quora.com/How-many-transistors-can-be-used-to-re...
But: "Synapses are usually separately modeled in transistors (they are not part of the neuron circuits described above) and dramatically add to the transistor count."
"[A neuron] has on average 7000 synaptic connections to other neurons."
So we're not there yet, but it's definitely an improvement!
I don't mean to suggest that transistors are equivalent to neurons or synapses; clearly they are very different things (though it is not clear that neurons are much more computationally powerful than transistors, despite assertions from many people that this is the case). But I think it is still useful to compare the complexity of structure of chips vs. the human brain. We are finally approaching the same order of magnitude of complexity in structure.
Also note that this is not manufactured on TSMC's highest density process. Assuming TSMC's 3nm process development is successful, that will probably be 6x denser than the 16nm process used here.