Exactly. Especially:

> ...the technique is already very computationally intensive, and the focus on individual neurons as a function of input means that they can't "reverse engineer" larger structures composed of multiple neurons nor a neuron that has multiple roles;

This paper just brings us no closer to explainability in black box neural networks and is just another excuse piece by OpenAI to try to please the explainability situation that has been missing for decades in neural networks.

It is also the reason why they cannot be trusted in the most serious of applications which such decision making requires lots of transparency rather than a model regurgitating nonsense confidently.

Like say, in court to detect if someone is lying? Or at an airport to detect drugs?

But, yes, those are also good examples of what we shouldn't be doing, but are going to do anyway.

Evolution is still doing it's thing.

Doesn't this criticism also apply to people to some extent? We don't know what the purpose of individual brain neurons is.

The machine model is not only a black box, but one incapable of understanding anything about its input, "thought process", or output. It will blindly spit out a response based on its training data and weights, without knowing the difference whether it true or false, meaningful or complete gibberish.

This seems like a novel approach to try to tackle the scale of the problem. Just because the earliest results aren’t great doesn’t mean it’s not a fruitful path to travel.

Is this true? I thought explainability for things like DNNs for vision made pretty good progress in the last decade.

The more interesting question is why are intelligence/beauty/consciousness emergent properties that exist in our minds.

Humans so far have done a great job at destroying nature faster than any other kind could.

And GPT4 was created for profit.

Perhaps intelligence is like a black box input to our bodies (call it the "soul", even though this isn't testable and therefore not a hypothesis). The mind therefore wouldn't play any more of a role in intelligence than the eye. And I'm not sure people would say the eye is necessary for understanding intelligence.

Now, I'm not really in a position to argue for such a thing, even if I believe it, but I'm curious what argument you might have against it.

Again, IANA neuroscientist, but this is my understanding from the literature and conversations with the scientists I work with.

Is it?

These networks are capable of copying something, yes. Do we have a good understanding of what that is?

Not really, no. At least I don’t. I’m sure lots of people have a much better understanding than I do, but I think its hard to know exactly whats going on.

People dismiss the stochastic parrot argument because of how impressive big neural nets are, but it doesn’t really invalidate that argument. Is a very, very, very good parrot that learns from everyone at once doing basically the same as what we do? I’d argue no, at least not fully. It’s absorbed aspects of us extremely well/is a very weird, sophisticated mirror, yes, and is copying something somehow, probably in a way reminiscent of how we copy. Is it basically the same as what we’re doing when we think? Partially? Fully? Not at all?

A typical engineer would say “good enough”. That type of response is valuable in a lot of contexts, but I think the willingness to apply it to these models is pretty reckless, even if it’s impossible to easily “prove” why.

To be clear on the exact statement you made, I think you’re right/it’s pretty clear neurons play some very important role/potentially capture a ton of what we consider intelligence, but I don’t think anyone really knows what exactly is being captured/what amount of thought and experience they’re responsible for.

Anyway, thank you for a great answer and conversation throughout this thread.

Regards neural networks, parroting and the emulation of intelligence (or the difference between an emulation and the "real thing"):

Well, somewhat like you say, we cannot propose a valid comparison from one to the other without an understanding of one (consciousness) or both. It's fascinating that there are some open, valid and pressing questions about what / how the output of this new wave of software is concretized (from foundational, semi-statistical algorithms in this case.)

Yes, I do agree neurons have something to do with the "final output". But this is a subsection of the problem - organic neurons is-an/are order(s) of magnitude in complexity beyond what the tricky "parrot" is up to. Moreso, these components perform very different functionally - the known functions of the neuron compared to ANN, backprop etc. The entire stack.)

P.S: One interesting theory I like to simulate and/or entertain is that every organic cell in the body has something to do with the final output of consciousness.

And yeah, are definitely a lot of open questions related to all of this. Love how its brought so many deep questions into focus.

It would explain why most "experts" didn't see GTP-4's abilities coming. Many of them expected that it would take a major algorithm or technology improvement to do "real intelligent" things, because they fundamentally misunderstood intelligence.

The effect of drugs (including alcohol) on the mind. Of note is anesthesia which can reliably and reversibly stop internal experience in the mind.

For a non-physical soul to hold our mind we would expect significant divergence from the above. Out of body experiences and similar are indistinguishable from dreams/hallucinations when tested against external reality (remote viewing and the like).

This sounds like it assumes a physical mind could access a non-physical soul. All we probably know is that we have to be using an intact mind to use free will.

That's not a completely watertight argument.

Consider a traditional FM/AM radio. You can modify it, damage it, and get notable changes to its behaviour...

> Of note is anesthesia which can reliably and reversibly stop internal experience in the mind

...turn it off and on again...

> For a non-physical soul to hold our mind we would expect significant divergence from the above.

... yet concluding that all the noises produced from the radio are purely internal, mechanical and physical would be the wrong conclusion.

(I'm not arguing that the human brain/mind is anything like analogous to a radio, just pointing out the limits of this approach.)

Although, sure, if we could somehow manage to determine that souls did exist then presumably an AI model as capable as a human would also be eligible for one.

Some people think that neurons specifically aren't necessary for understanding intelligence but in the same way that understanding transistors isn't necessary to understand computers, that neurons comprise the units that more readily explain intelligence.

That’s not necessarily a disproof. It’s also not necessarily reasonable to conflate what we call “the soul” with intelligence.

This is entering the world of philosophy, metaphysics and religion and leaving the world of science.

The modern instinct is to simply call bullshit on anything which cannot be materially verified, which is in many ways a very wise thing to do. But it’s worth leaving a door open for weirdness, because apart from very limited kinds of mathematical truth (maybe), I think everything we’ve ever thought to be true has had deeper layers revealed to us than we could have previously imagined.

Consider the reported experience of people who’ve had strokes and lost their ability to speak, and then later regained that ability through therapy. They report experiencing their own thoughts and wanting to speak, but something goes wrong/they can’t translate that into a physical manifestation of their inner state.

Emotional regulation, personality, memory, processing speed, etc… are those really that different from speech? Are they really the essence of who we are, or are they a bridge to the physical world manifest within our bodies?

We can’t reverse most brain damage, so it’s usually not possible to ask a person what their experience of a damaged state is like in comparison to an improved state. We do have a rough, strange kind of comparison in thinking about our younger selves, though. We were all previously pre memory, drooling, poorly regulated babies (and before that, fetuses with no real perception at all). Is it right to say you didn’t have a soul when you were 3 weeks? A year? Two years? When exactly does “you” begin? I can’t remember who I was when I was when I was 2 months with any clarity at all, and you could certainly put different babies in doctored videos and I wouldn’t be able to tell what was me/make up stories and I’d probably just absorb them. But I’m still me and am that 2 month old, much later in time. Whatever I’m experiencing has a weird kind of continuity. Is that encoded in the brain, even though I can’t remember it? Almost definitely, yeah. Is that all of what that experience of continuity is, and where that sense is coming from? I’ve got no idea. I certainly feel deeper. Remember that we all are not living in the real world, we’re all living in our conscious perception. The notion that we can see all of it within a conscious mirror is a pretty bold claim. We can see a lot of it and damage the bridge/poke ourselves in the eyes with icepicks and whatnot, and that does stuff, but what exactly is it doing? Can we really know?

Intuitively most people would say they were still themselves when they were babies despite the lack of physical development of the brain. Whatever is constructing that continuous experience of self is not memory, because that’s not always there, not intelligence, because that’s not always there, not personality, because that’s not always there… it’s weird.

I think it’s important to remember that. Whenever people think they have human beings fully figured out down to the last mechanical detail and have sufficient understanding to declare who does and doesn’t have a soul and what that means in physical terms, bad things tend to happen. And that goes beyond a statement to be cautious about this kind of stuff purely out of moral hazard; the continual hazard is always as empirical as it is moral. We can never really know what we are. Our perceptual limitations may prove assumptions we make about what we are to be terribly, terribly wrong, despite what seems like certainty.

Suppose a soul is an immaterial source of intelligence, but it controls the body via machine-like material hardware such as neurons.

Or an alternative, suppose there is a soul inside your body “watching” the sensory activations within your brain like a movie. The brain and body create the movie & have some intelligence, but other important properties of the consciousness are bound to this observer entity.

In both these cases, the test just shows that if you damage the hardware, you can no longer observe intelligence because you’ve broken the end-to-end flow of the machine.

But yes, supposing that then you would expect to only see damages that correspond such as different forms of paralysis or other purely mechanical damages, not things that change the interior perspective.

Otherwise you start postulating the existence of a thing whose sole justification is your desire for the existence of that thing, which is natural when you start questioning beliefs and kick out all the supports without meaning to.

I think this is what Bertrand Russel's teapot was meant to ellucidate.

But the same is true for an ant.

From the New Yorker [1]:

> The researchers aren’t yet sure exactly what causes the digestive slowdown. Maybe it’s stress. Maybe the fluid that ants exchange when they share food stimulates digestion. Maybe they die still looking for a relative to break bread with.

[1] https://www.newyorker.com/tech/annals-of-technology/lonely-a...

Ants have developed architecture, with plumbing, ventilation, nurseries for rearing the young, and paved thoroughfares. Ants practice agriculture, including animal husbandry. Ants have social stratification that differs from but is comparable to that of human cultures, with division of labor into worker, soldier, and other specialties that do not have a clear human analogy.

Ants enslave other ants. Ants interactively teach other ants, something few other animals do, among them humans. Ants have built "supercolonies" dwarfing any human city, stretching over 5,000 km in one place. And ants too have a complex culture of sorts, including rich languages based on pheromones.

Despite the radically different nature of our two civilizations, it is undeniable from an objective standpoint that this level of society has been achieved by ants.

[0]: https://www.reddit.com/r/unpopularopinion/comments/t2h1vs/an...

I could go on and on. Point being, analogies are fun and sometimes illuminating, but they're just that. There's a vast difference in complexity between what ants do, and what humans do.

They're saying they accomplish incredible things for the size of their brain, which is absolutely and unequivocally true.

"Go to the ant, thou sluggard; consider her ways, and be wise".

Interesting parallell with intelligence/sentience/sapience. Despite the means, isn't the end result what you have to judge? The end result looks like a rudimentary civilization. How much back in time would we have to go back to find more sophistication in ant societies than humans?

I'm indeed talking about the ends. The operative phrase here is "it looks like". It looks like a civilization, there are even some structural similarities to be found - but it is not a civilization, anymore than a Lego brick with a computer sticker on it is a PC.

This is not to deny that ants are awesome and very much worth studying - my point is to avoid following flawed analogies too far. "Ant colonies look like rudimentary civilization, and civilization is/does X, therefore..." is going to be a nonsense statements for most of possible values of X.

Ants can do it as a consequence of their size. Relative lifting strength drops fast with increased size. Conversely, an ant scaled to human size would collapse under its own weight and cook itself to death - waste heat generation scales with volume (~ size³), while waste heat rejection scales with surface area (~ size²).

And it ain't a cognitive achievement anyway.

> Humans can't communicate in real-time with pheromones alone.

Yes. Because it's not as useful at our scale, nor is it real-time - chemical communication works better for small organisms and small volumes of living, as the travel speed and dissipation rate of pheromones is independent of organisms emitting them. Meanwhile, we have multiple ways of communicating real-time, some of which work at light speed which is the definition of "real time" (light speed in vacuum being the speed of causality itself).

> Most humans do not know how to build their own home.

Neither do ants.

> An ant might well consider us backwards, not advanced.

An ant can't consider us anything. The point I'm trying to get across is, just because the ant colony is capable of surprisingly high sophistication, doesn't mean the individual ants are.

As a counterpoint that's actually cognitive in nature: AFAIK individuals of many (most?) ant species can be easily tricked into following each other in a circle, and they will continue to do so until they start dying from starvation. This is because the ant isn't making a decision following a complex thought process - it's executing a simple algorithm, that works well in nature because nature is chaotic enough that stable ant circles are unlikely to form (and when they do, they're unlikely to be left undisturbed for long).

To give some examples of ant agriculture and animal husbandry, they cut and feed leaves to a fungus they farm for food. The fungus even communicates to the ants to switch to a different plant when the plant produces toxins in defense to cutting.

Ants herd aphids, protecting them from predators (ladybugs), secrete pheromones that direct and pacify them, and massage their abdomens to milk sugar dew.

1. Neurons connect all our senses and all our muscles.

2. Neurons are the definitive difference between the brain and the rest of the body. There is “other stuff” in the brain, but it’s not so different from the “other stuff” that’s in your rear end.

Don’t underestimate what a neuron can do. A single artificial neuron can fit a logistic regression model. A quarter of a million is on then scale of some our our largest AI, and biological neurons are far more connected than ANN. An ant quite likely has a more powerful brain than GPT-4.

Our digestive systems appear to be important to our behaviour, though. Some recent work in mice showed that if colonised with bacteria from faeces of humans with autism, the mice would begin to show autistic behaviours.

So, not sure your argument here is especially strong.

If the study you mention show something else, I would actually be very interested in a source.

The paper is somewhat new so I haven't done a proper review to know if it's solid work yet, but this may offer some context for some of the comments in this thread.

https://pub.towardsai.net/ais-mind-reading-revolution-how-gp...

so why not have them decode sequential dense vectors of their own activations?

As for the majority scoring poorly, they suggest that most neurons won't have clear activation semantics so that is intrinsic to the task and you'd have to move to "decoding the semantics of neurons that fire as a group"

I wonder how often this happens in the universe...

Is that evident already or are we fitting the definition of intelligence without being aware?

I don't think there's a single dollar of funding in AI right now going to the "problem" of making an AI that has terminal preferences, sets instrumental goals to achieve those preferences, iteratively observes the environment and makes predictions of how well its actions will help optimize the future toward its preferences, etc.

Though, interestingly, it might not even be that hard a problem, compared to all the problems being solved in AI right now. A fruit fly is already a goal-directed agent in the sense described above. Yet a fruit fly has only 200K neurons, and very few of the connections between those neurons are dynamic; most are "hard wired" by [probably] genetics.

If we want true ALife, we only need to understand what a fruit fly brain is doing, and then model it. And that model will then fit — with room to spare! — on a single GPU. From a decade ago.

The entire (enormous) field of reinforcement learning begs to differ.

Yes, and put that compact fruit fly in silico brain into my Roomba please so that it does not get stuck under the bed.

This is the kind of embodied AI that should really worry us.

Don’t we all suspect deep skunkworks “defense” projects of these types?

Computationally, a biological neuron is equivalent to something like an entire artificial neural net.

(Notice how baseless comments can sway either way)

No they can’t! ;)

Even performing a map-reduce over large documents to summarize or analyze them for a specific audience is largely beyond it. A 32K context size is a pittance when it comes to a single Title in the USC or CFR, which average into the millions of tokens each.

How do you think people become lawyers and how smart do you actually think the average lawyer is? The problem is that there's hundreds of thousands if not millions of pages, not that it requires superhuman intelligence to understand.

Even if it were capable of intelligence in the bottom quartile of humanity it would be SO MUCH more useful than it is now because I'd be able run and get something useful out of thousands of models in parallel. As it stands now GPT4 fails miserably at scaling up the kind of reasoning and understanding that even relatively stupid humans are capable of.

[1] https://www.amazon.com/Introduction-Legal-Reasoning-Edward-L...

It quickly breaks down even at 8k with legislation that is even remotely nontrivial.

What if someone finds a good practical way to expand the context length to 10M tokens? Do you think such model won't be able to do your task?

It seems like you have an opportunity to compare 8k and 32k GPT-4 variants (I don't) - do you notice the difference?

I can mow my lawn yet I still hire landscapers. That doesn't say anything about the difficulty of cutting grass or the intelligence of a DeWalt lawnmower but about specialization and economic tradeoffs - like the liability insurance accountants carry for their client work.

> What if someone finds a good practical way to expand the context length to 10M tokens? Do you think such model won't be able to do your task?

Not based on the transformer architecture (aka predict next token). It already fails at most of my use cases at 32K by default, unless I go to great lengths to tune the prompt.

> It seems like you have an opportunity to compare 8k and 32k GPT-4 variants (I don't) - do you notice the difference?

32K works better for my use case but requires much more careful prompt "engineering" to keep it from going off the rails. In practice, actually getting full 32K use out of it is a disaster since the connection will drop and I have to resend the entire context with a "continue" message, costing upwards of $10 for what should cost $2-4 per call. I haven't actually tried 32K on as much as a whole USC Title because that would costs thousands.

That would work.

I'm not sure that pointing out that LLMs are as useful for parsing legal code as the average human is something to brag about though.

All we've shown is that LLMs are as useful for parsing legal text as the average human. Which is to say, not.

A dog is also as useful for parsing legal texts as the average human. So is a rock.

First hand experience -I’ve been using it daily for the past two months.

I've used Google search for decades and it's been able to answer questions better than humans ever could. Same for Google Maps, though arguably they're the same system at this point. My calculator is far more intelligent than any human I've met, at least when it comes to adding large numbers. My compiler can detect even the slightest syntax error with impeccable accuracy. Microsoft word has an incredible vocabulary. Wikipedia knows more historical events than any human dead or alive. And so on.

Shit, users thought Eliza was intelligent in the 60s.

If what you really mean is that LLMs are cool and useful, then sure. Just say that instead of couching it in some vague claim of intelligence.

When you say "intelligent", what do you mean exactly?

What might help is describing what specific interactions give you the impression that GPT-4 is intelligent?

My interactions with GPT-4 include quite a wide range of questions: "how to talk to my kid in this specific situation", "what could have happened if Germany had won WW2", "what does this code do", "here's an idea for a research paper, let's brainstorm the details and implementation". I can also discuss with it anything that's been written in this thread and I'm sure it would provide intelligent responses (I haven't, btw).

If you can't enjoy it, is it worth it? Do AI's experience joy?

I have no idea.

By that definition, every computer virus and worm qualifies as having "self-directed thought" and "agency." Their very existence "to keep running" and propagate satisfies the need "to interact with the world."

LLM's do not think.

LLM's are algorithms.

I personally wouldn’t rule out that it can’t be emulated in a different substrate, but I think calling it “an algorithm” is to def stretch and misapply the usefulness of the term.

The only way our brains could be not algorithmic is if something like soul is a real thing that actually drives our intelligence.

Therein lies the question, one which deserves contemplation and can lead to Enlightenment.

Which then begs the question; is Enlightenment a "real thing" and, if not, how is it that it can be experienced?

Edit: I probably should define reasoning as solely “deductive reasoning”, in which case, perhaps it is better than humans. But that seems like a premature claim. On the other hand, non-deductive reasoning, I have yet to see from it. I personally can’t imagine how it could do so reliably (from a human perspective) without real-world experiences and perceptions. I’m the sort that believes a true AGI would require a highly-perceptual, space-occupying organ. In other words it would have to be and “feel” embodied, in time and space, in order to perform other forms of reasoning.

Not sure an example is needed because I agree it “explains” better than pretty much everyone. (From my mostly lay perspective) It essentially uses your prompt as an argument in a probabilistic analysis of its incredibly vast store of prior inputs to transform them into an output that satisfies your prompt. This is cool and useful, to say the least. But this is only one kind of reasoning. A machine without embodied perceptual experiences simply cannot reason to the full-extent of a human.

As detailed as possible, describe what happened.

I’m sure it elicited ideas in your head based on your own experiences. You could then use those ideas to ask questions and get further information. Or you could simply pick an answer and then delve into all the details and sensations involved, creating a story based on what you know about the world and the feelings you’ve had.

I could have created a more involved “prompt story” one with more details but still somewhat vague. You would probably have either jumped straight to a conclusion about what happened or asked further questions.

Something like “He kicked a ball at my face and hit me in the nose. I laughed. He cried.”

Again, vague. But if you’ve been in such a situation you might have a good guess as to what happened and how it felt to the participants. ChatGPT would have no idea whatsoever as it has no feelings of its own with which to begin a guess.

Consider poetry. How can ChatGPT reason about poetry? Poetry is about creating feeling. The content is often beside the point. Many humans “fail” at understanding poetry, especially children, but there are of course many humans that “get it”, escpecially after building up enough life experience. ChatGPT could never get it.

Likewise for psychedelic or spiritual experiences. One can’t explain such experience to one who has never had it and ChatGPT will never have it.

Same goes for all inner experience.

GPT-4 has a lot of such fake memories. It knows a lot about the world, and about feelings, because it has "experienced" a lot of detailed descriptions of all kinds of sensations. Far more than any human has actually experienced in their lifetime. If you can express it in words, be it poetry, or otherwise, GPT-4 can understand it and reason about it, just as well as most humans. Its training data is equivalent to millions of life experiences, and it is already at the scale where it might be capable of absorbing more of these experiences than any individual human.

GPT-4 does not "get" poetry in the same way a human does, but it can describe very well the feelings a human is likely to feel when reading any particular piece of poetry. You don't need to explain such things to GPT-4 - it already knows, probably a lot more than you do. At least in any testable way.

ChatGPT is a machine, an algorithm, a recombinator of symbols. It doesn’t know what the symbols refer to because each symbol necessarily refers to another symbol until you finally reach a symbol that refers to a shared, real experience…perhaps (Hello Wittgenstein!). And ChatGPT has no experience. Just symbols. It can’t intuit anything. It can’t feel anything. Even if you put quotes around “feel”, what does that even mean for a software algorithm running on hardware that does not feed continuous, variable electrical sensations to the algorithm? It only feeds discrete symbols. Do you feel the number 739? Or do you “feel” it? Um what? Whatever inner experience 739 happens to produce in you is grounded in some real experiences in the past. Likewise any fake memories you have that somehow seem real, those are still grounded in a real feelings at some point. You could do this ad infinitum. If you are alive, you have experience. But ChatGPT has no experience, no grounding.

Problem here might be that we are trying to use words and logic to describe something that cannot be described by either.

This is why the gong is struck.

GPT-4 will often come up with a solution to a problem, but only if it has learnt something similar (it's better than Google in some respects: it can extract and combine abstractions).

However, both need handholding by a human (supplying the initiative and directing around mistakes).

If GPT-4 can't intuit an answer then it just goes in circles. It can't reason its way through a novel problem. If you start questioning it then it's clear that it doesn't understand what it's doing.

It might be a stepping stone towards AGI, but I'm a bit bemused by anyone claiming that it has anything like the reasoning skills of a human. That is far from the impression I get, even though I find it a useful tool.

One. Token.

One token, that is not an answer to your question, that is not asking for clarifications, that is not trying to reason about your question, or how to answer it, because it's just one token.

One. Token.

For example, say you ask:

  "What is the sieve of Eratosthenes? Speak in chicken".

  Chicken

  What is the sieve of Eratosthenes? Speak in chicken. Chicken

  chicken

  Iteration 2: What is the sieve of Eratosthenes? Speak in chicken. Chicken chicken

  ... 

  Iteration k: What is the sieve of Eratosthenes? Speak in chicken. Chicken chicken chicken chicken chicken chicken chicken chicken chicken chicken ...

And do you know when the chickens stop? When GPT-4 generates a special and magickal token, called a stop-token (or a "stop sequence" in OpenAI docs). That's a token, not found in the training corpus, added to the end of every string during tokenisation. That's how ChatGPT knows to stop sending back your prompt + generated tokens, to the model. It can't look back to what GPT-4 has generated so far, because it doesn't understand any of that. Because it doesn't understand anything, and therefore cannot reason about your question, or realise it has answered it. It cannot do anything except a) ask GPT-4 to generate another token or b) stop asking for more tokens.

  "What is the sieve of Eratosthenes? Speak in chicken". Chicken, chicken, chicken, chicken, .... chicken, stop_token!

And that's how GPT-4 explains what the Seive of Eratosthenes is, but in chicken.

So what you see as a user is like watching a movie where a plate of spaghetti is flying through the air, lands on a table cloth, the tablecloth lands on a table, the table lands on the floor, four chairs land around it and suddendly there's people in the chairs eating the spaghetti. It's not that someone has figured out how to reverse time: it's a recording, played backwards. It looks like things are going backwards, but they're not.

It looks like ChatGPT is answering your questions, but it's not. It looks like ChatGPT is interacting with you, but it's not. It's a magick trick.

Answer from GPT-4:

Sure, here's the explanation of the sieve of Eratosthenes explained in a simplified "chicken" style:

Imagine you are a chicken. You have lots of chicken friends. Let's say there are 100 of you. Now, you all stand in a line. Each chicken is numbered, from 2 up to 100. Chicken number 1 decided to stay at home because he's a bit special in this game.

Chicken number 2, a very important chicken, starts to crow. Every chicken in line after number 2 who has a number that is a multiple of 2 (so every second chicken from chicken number 2) gets scared and runs away.

Then chicken number 3, the next chicken in line who didn't run away, starts to crow. Every chicken in line after number 3 who has a number that is a multiple of 3 (so every third chicken from chicken number 3) also gets scared and runs away.

This continues with the next chicken in line who didn't run away, and so on. Each chicken scares away all the chickens whose numbers are multiples of their own number.

At the end, the chickens that are still in line, who didn't run away, are very special chickens. Their numbers are what humans call prime numbers. These are numbers that can only be divided by 1 and themselves.

This whole chicken game is like the Sieve of Eratosthenes. It's a way that humans have figured out to find prime numbers. You start from the lowest number (2) and eliminate multiples of each number. The numbers that are left over are the primes.

And that, in chicken speak, is the Sieve of Eratosthenes!

Answer from GPT-4:

The chicken is in a chicken coop with many other chickens. Each chicken represents a number in the chicken coop. The first chicken that hasn't been touched, after the chicken representing one, is a prime chicken. We'll call this chicken the prime chicken.

Now, imagine that the prime chicken has a unique chicken dance. Every chicken that copies this prime chicken's dance is removed from the coop because they are not prime chickens. These are the chickens that would be multiples of our prime chicken.

Then we find the next chicken that hasn't been touched and doesn't know the dance of the previous prime chicken. This chicken is also a prime chicken. This prime chicken has a different chicken dance, and all the chickens that copy this new dance are also removed from the coop.

We keep doing this - finding the next chicken that hasn't danced yet, designating it as a new prime chicken with its unique dance, and removing any chickens that copy the dance - until there are no more new dances.

All the chickens left in the coop, who haven't copied any dances, are the prime chickens. This is the chicken way to understand the Sieve of Eratosthenes. The prime chickens are like the prime numbers in the Sieve of Eratosthenes.

----

I then followed up with: "You must only use the word "chicken". Do not use any other words."

Then it wrote an entire page filled with "chicken chicken chicken"

I then asked: "By what logic did you construct your response? Why not use just one occurrence of the word "chicken" and finish there?"

Response from GPT-4:

Apologies for any confusion. Your question seemed to request a description of the Sieve of Eratosthenes using only the word "chicken". While it's true that a single "chicken" would technically meet your request, it seemed that you were interested in a more elaborate or extended response, hence the repetition.

In any case, attempting to convey any meaningful information using only a single word—let alone a complex concept like the Sieve of Eratosthenes—is inherently impossible due to the lack of distinct symbols or constructs to represent different aspects of the concept. Thus, the extended repetition of "chicken" was intended more as a playful response rather than a serious attempt at conveying the concept you requested.

False. The neural network inside the transformer LLM contains a hierarchical semantic model, and has inferred some rules of reasoning from the training set. It can apply those rules to new input.

There are semantic layers above the "token token tokens".

Explore them here: https://openaipublic.blob.core.windows.net/neuron-explainer/...

If you are not aware of the long history of debunking such fabrications, I suggest you start here:

Connectionism and Cognitive Architecture: A Critical Analysis

https://ruccs.rutgers.edu/images/personal-zenon-pylyshyn/pro...

https://en.wikipedia.org/wiki/Artificial_neuron

These models infer semantic categories that correlate to categories within the human mind, to the extent that they can solve natural language understanding tasks.

No one is saying they are biological neurons, or that they model semantics exactly as the human mind would. It is mechanical pattern recognition that approximates our understanding.

You can browse those artificial neurons online and view their associations.

For example:

>> It is mechanical pattern recognition that approximates our understanding.

That's just a claim and you're not even saying why you make it, what makes you think so, etc.

Mechanical - it is an algorithm, not a living being.

Pattern recognition - a branch of machine learning that focuses on the detection and identification of regularities and patterns in data. It involves classifying or categorizing input data into identifiable classes based on extracted features. The patterns recognized could be in various forms, such as visual patterns, speech patterns, or patterns in text data.

Approximates our understanding - meaning the model is not exactly the same as human understanding

When I say 'mechanical pattern recognition that approximates our understanding,' what I mean is that large language models (LLMs) like GPT-4 learn patterns from the vast amounts of text data they're trained on. These patterns correspond to various aspects of language and meaning.

For example, the models learn that the word 'cat' often appears in contexts related to animals, pets, and felines, and they learn that it's often associated with words like 'meow' or 'fur'. In this sense, the model 'understands' the concept of a cat to the extent that it can accurately predict and generate text about cats based on the patterns it has learned.

This isn't the same as human understanding, of course. Humans understand cats as living creatures with certain behaviors and physical characteristics, and we have personal experiences and emotions associated with cats. A language model doesn't have any of this - its 'understanding' is purely statistical and based on text patterns.

The evidence for these claims comes from the performance of these models on various tasks. They can generate coherent, contextually appropriate text, and they can answer questions, translate languages, and perform other language-related tasks with a high degree of accuracy. All of this suggests that they have learned meaningful patterns from their training data.

I hope this provides more context and supports my previous statements. I understand this is a complex topic, and I appreciate your challenging questions as they help clarify and deepen the discussion.

I have a suggestion: try to convince yourself that you are wrong; not right. Science gives you the tools to know when you're wrong. If you're certain you're right about something then you're probably wrong and you should keep searching until you find where and how.

For example, try to trace in your mind the mechanisms and functionality of language models, and see where your assumptions about their abilities come from.

Good luck.

Let's delve deeper into the mechanics of language models. Large language models like GPT-4 use an architecture called transformers. This architecture is composed of layers of self-attention mechanisms, which allow the model to weigh the importance of each word in the input when predicting the next word.

When the model is trained, it adjusts the weights in its network to minimize the difference between its predictions and the actual words in its training data. This process is guided by a loss function and an optimization algorithm.

Through this training process, the model learns to represent words and phrases as high-dimensional vectors, also known as embeddings. These embeddings capture many aspects of the words' meanings, including their syntactic roles and their semantic similarities to other words.

When the model generates text, it uses these embeddings to choose the most likely next word given the previous words. This process is based on the patterns and regularities that the model has learned from its training data.

Of course, this is a high-level description and the actual process involves a lot of complex mathematics and computation. But I hope it gives you a better sense of the mechanisms behind these models.

As for evidence, there are numerous studies that have evaluated these models on a wide range of tasks, including text generation, question answering, translation, and more. These studies consistently show that these models perform well on these tasks, often achieving state-of-the-art results. This is empirical evidence that supports the claim that these models have learned meaningful patterns from their training data.

I agree that we should always remain skeptical and open to new evidence and alternative explanations. I welcome any specific criticisms or alternative hypotheses you might have about these models and their capabilities.

For the record, I just polished off a PhD in AI (symbolic machine learning) after a Master's where I studied neural nets and NLP, including plenty of language generation. You're teaching your grandma to suck eggs.

And I'm really very tired with this kind of conversation that never teaches me anything new. Your comment is still "what"'s all the way down. You never explain why or how word embeddings capture aspects of meaning, you 're just repeating the claims by Mikolov or whoever. Look, here:

>> Through this training process, the model learns to represent words and phrases as high-dimensional vectors, also known as embeddings. These embeddings capture many aspects of the words' meanings, including their syntactic roles and their semantic similarities to other words.

That's just a claim, made long ago, and challenged at the time, and the challenge ignored, and it keeps being bandied about as some kind of scientific truth just because critics got tired or bored having their criticims consistently ignored and gave up trying.

This is what I point out above: connectionists never stop to consider criticism of the limitations of their systems' until someone rubs their face in it - like Minsky and Pappert did in 1969, which then caused them to be forever reviled and accused of causing an AI winter, when what they really caused was some connectionists to get off their butts and try to improve their work, a process without which we wouldn't, today, have backpropagation applied to NNs, and the potent image classifiers, good models of text, etc, that it enabled.

As to the "evidence" you profer, mainly preprints on arxiv, and mainly consisting of budding researchers uploading papers consisting of little more than leaderboards (those little tables with the systems on one side, the datasets on the other side, and your system's results in bold, or no paper) those are useless. 99% of research output in deep learning and neural nets is completely useless and never goes anywhere- because it lacks novelty, it is completely devoid of any theoretical results, and it is unreproducible even when the code is available.

For example, you mention studies on "question answering". Ca. 2018 Google published a paper where they reported that their BERT language model scored near-human performance on some question answering dataset without ever even having been trained on question answering. A scientific miracle! Some boffins who clearly don't believe in miracles wondered why that would even be possible and dug a bit, and found that BERT was overfitting to surface statisical regularities of its dataset. They created a new test dataset devoid of such statistical regularities and BERT's performance went down the drain, until it hit rock bottom (a.k.a. "no better than chance"). So much for "semantic similarity" measured over word embeddings modelling meaning.

But this is exactly the kind of work that I say connectionists consistently ignore: nowhere will you find that subsequent language models were tested in the same way. You will instead find plenty of tests "demonstrating" the ability of language models to represent semantics, meaning, etc. It's all bullshit, self-delusion at best, conscious fabrications otherwise.

This is the paper (I'm not affiliated with it in any way):

Probing Neural Network Comprehension of Natural Language Arguments

https://aclanthology.org/P19-1459/

But this kind of work is thankless for the undertaking academics and most of us have more important things to do. So the criticism eventually dwindles and what remains is the bullshit, and the fabrications, and the fantasies, seeping into mainstream discourse and being repeated uncritically - by yourself, for example. I can't even summon the compassion to not blame you anymore. For all I know you're exactly one of those connectionists who don't even understand their work is not science anymore, but spectacle.

P.S. I am not blind to the change of tone in your recent comments and I'm really sorry to be so cranky in response, when I should be cordial in reciprocity, but I've really had enough of all this. The unscientific bullshit that has permeated everything is now making it harder for me to fund my own research, just because I'm not prepared to make ridiculous claims about its intelligence or understanding. You know why I'm not? Because it's just maths. Soon nobody will be able to get even basic CS research funded unless they include a section explaining why they don't use OpenAI's chatbots instead.

Semantics in word embeddings: While I agree that word embeddings cannot fully capture human-like semantic understanding, they do provide a mathematical representation that has proven useful in many NLP tasks. It's not that word embeddings "understand" semantics in the human sense, but they do capture certain aspects of meaning that are statistically derived from their use in the training corpus. This is not an unsubstantiated claim. It is empirically demonstrated in numerous tasks where semantic understanding is beneficial, like semantic similarity, word analogy, and other downstream tasks such as sentiment analysis, text classification, etc.

You mentioned that connectionists only improve their systems when someone rubs their face in it. This is essentially how scientific progress happens - through skepticism, criticism, and the relentless pursuit of truth. I would argue that the current era of deep learning research is no different. It's a messy, iterative process, with steps forward, backward, and sideways.

Your point about BERT overfitting to statistical regularities of the dataset is well taken. Indeed, it exposes the limitations of the model and the need for careful design and evaluation of benchmarks. However, it's worth noting that a failure in one specific test doesn't invalidate the successes in other tasks. It simply highlights an area that needs improvement.

It's true that there's a flood of papers and not all of them have substantial novelty or impact. This is not a problem exclusive to deep learning or AI, but a broader issue in academia and scientific publishing. However, amidst the noise, there's also a lot of valuable work being done, with genuine advancements and novel approaches.

Finally, while it's true that we may be over-optimistic or uncritical at times, it's also important to recognize the significant advancements that have been made. AI systems, though imperfect, are now capable of tasks that were thought to be infeasible not too long ago. That's not a spectacle, but a testament to the collective efforts of researchers around the world.

I appreciate your critical perspective and your emphasis on rigorous scientific inquiry. Such discussions are important for the health and progress of the field.

PS: I understand where you're coming from. Better to not take any of it personally. Sometimes I need a break from this too. Remember there is no malicious intent here when people are just sharing their views.

While AI doesn't have an innate understanding of the world as humans do, the semantic representations it learns from vast amounts of text data can be surprisingly rich and detailed. It can capture associations and nuances that are not immediately apparent from a purely syntactic analysis of the text.

And note you're still not saying "why" or "how", only repeating the "what" of someone else's claim.

Semantic similarity methods are based on the idea that the meaning of a word can be inferred from its context, which is a concept known as distributional semantics. In essence, words that occur in similar contexts tend to have similar meanings. This is not just a heuristic, it's a well-established principle in linguistics, known as the distributional hypothesis.

In the case of large language models, they are trained on vast amounts of text data and learn to predict the next word in a sentence given the previous words. Through this process, they learn to represent words as high-dimensional vectors (word embeddings) that capture many aspects of their meaning, including their semantic similarity to other words.

These models can generate coherent text, answer questions, translate languages, and perform other language-related tasks with a high degree of accuracy. These capabilities wouldn't be possible if the models were only capturing syntax and not semantics.

The 'why' is because these models learn from the statistical regularities in their training data, which encode both syntactic and semantic information. The 'how' is through the use of deep learning algorithms and architectures like transformers, which allow the models to capture complex patterns and relationships in the data.

I hope this provides a more detailed explanation of my argument. I'm not trying to 'pull rank', but simply explaining the basis for my claims. I understand this is a complex topic, and I appreciate your challenging questions as they help clarify and deepen the discussion.

This makes as much sense as me asserting a binary search can tell one why they are looking for what they want.

That is to say, it cannot.

Even for obviously nonsens that 90% of humans would recognice as such.

There are many types of reasoning, but GPT-4 gets 97% on casual discovery, and 92% on counterfactuals (only 6% off from human, btw) with 86% on actual causality benchmarks.

I'm not sure yet if the question is correct, or even appropriate/achievable to what many may want to ask (i.e. what 'the public's is interested in is typically lost after it is defined in any given study); however this is one of the best works available to address this problem I've seen so far, so perhaps it can help.

Remember that GPT is not trained on all possible text. It's trained on text that was written intentionally. What percentage of that text contains "correct" instances of causal discovery, counterfactuals, etc.?

EDIT: Seems so...

https://duckduckgo.com/?q=ESTIMATE+OF+GPT-4%27S+IQ&t=opera&i...

shows articles with GPT IQ from 114 to 130. Change is coming for humans.

Me: A room contains a blue box and a red box. Alice stands on the red box. Bob also stands on a box. No box has multiple people standing on it. Which box is Bob stood on?

ChatGPT: There are two boxes in the room: a blue box and a red box. Alice is standing on the red box, so Bob must be standing on the blue box. Therefore, Bob is standing on the blue box.

Me: Add a green box to the room. Which box is bob standing on?

ChatGPT: If a green box is added to the room, then there are now three boxes: a blue box, a red box, and a green box. Since Alice is still standing on the red box, we know that Bob must be standing on either the blue box or the green box. We cannot determine which one Bob is standing on without additional information.

Think of business / artistic / cultural leaders nurturing protégés despite not totally understanding why they’re successful.

Of course those protégés have agency and drive, so maybe not a perfect analogy. But I’m going to stand by the point intuitively even if a better example escapes me.

An example: we are not very good at creating flight, the one birds do and humans always regarded as flight, and yet we fly across half the globe in one day.

Going up three meters and landing on a branch is a different matter.

In human brains, language is only a way to communicate thoughts in concept form, though we also seem to use language to communicate abstract thoughts to ourselves to break them apart/down in a way (imo).

I'd love to see someone train a model on the level of GPT4 to generate abstract thoughts/ideas based on input/context and then pair this model with GPT4 co-operatively and continue to train, such that the flow of abstract ideas is parsed by GPT. But like...how do you even train a model that operates on abstract ideas, there doesn't seem to be any way to do this.

Natural language understanding comprises a wide range of diverse tasks such as textual entailment, question answering, semantic similarity assessment, and document classification. Although large unlabeled text corpora are abundant, labeled data for learning these specific tasks is scarce, making it challenging for discriminatively trained models to perform adequately. We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task. In contrast to previous approaches, we make use of task-aware input transformations during fine-tuning to achieve effective transfer while requiring minimal changes to the model architecture. We demonstrate the effectiveness of our approach on a wide range of benchmarks for natural language understanding. Our general task-agnostic model outperforms discriminatively trained models that use architectures specifically crafted for each task, significantly improving upon the state of the art in 9 out of the 12 tasks studied. For instance, we achieve absolute improvements of 8.9% on commonsense reasoning (Stories Cloze Test), 5.7% on question answering (RACE), and 1.5% on textual entailment (MultiNLI).

Should that be measured in number of nuclear power plants needed to run the computation? Or like, fractions of a small star’s output?