A lot of people didn't seem to get it when it was discussed on HN. A GPT had _only_ ever seen Othello transripts like: "E3, D3, C4 ..." and NOTHING else. It knows nothing of the board. It doesnt event know that there are two players. It learned Othello like it was a language, and was able to play an OK game of it, making legal moves 99.99% of the time. Inside its 'mind', by looking for correlations between its internal state and what they knew the 'board' would look like at each step in the games, they found 64 nodes that seemed to represent the 8x8 Othello board and representation of the two different colours of counters.

And this is the key bit: They reached into its mind and flipped bits on that internal representation (to change white pieces to black for example) and it responded in the appropriate way when making the next move. And by doing this they were able to map out its internal model in more detail, by running again and again with different variations of each move.

Yeah, like language have gramma rules games also have rules, in both cases LLM can learn rules, it's the same with many other structured chains of actions/tokens, you could also model actions from different domains and use them as language. It seems a lot of emergent behaviours of LLMs are what you could call generalized approximated algorithms for certain tasks. If we could distill only these patterns and extract them and maybe understand them (if possible, as some of these are HUGE) then based on this knowledge maybe we could create traditional algorithms that would solve similar problems.

As much as I want to be enthusiastic about this, it’s not entirely clear to me that it is surprising that such a feat can be achieved. For example it may be possible to train a 2 layer MLP to predict the state of a tile directly from the inputs. It may be that the most influential activations are closer to the inputs then the outputs, implying that Othello-GPT itself doesn’t have a world model, instead showing that you can predict board colors from the transcript. Again, not a practitioner but once you are indirecting internal state through a 2 layer MLP it gets less obvious to me that the world model is really there. I think it would be more impressive if they were only taking “later” activations (further from the input), and using a linear classifier to ensure the world model isn’t in the tile predictor instead of Othello-GPT. I would appreciate it if somebody could illuminate or set my admittedly naive intuitions straight!

That said, I am reminded of another OpenAI paper [1] from way back in 2017 that blew my mind. Unsupervised “predict the next character” training on 82 million amazon reviews, then use the activations to train a linear classifier to predict sentiment. And it turns out they find a single neuron activation is responsible for the bulk of the sentiment!

[1] https://openai.com/blog/unsupervised-sentiment-neuron/

A “Classic” neural network, where every node from layer i is connected to every node on layer i+1

It turns out that the error rates of these probes are reduced from 26.2% on a randomly-initialized Othello-GPT to only 1.7% on a trained Othello-GPT. This suggests that there exists a world model in the internal representation of a trained Othello-GPT.

I take that to mean that the 64 trained Probes are then shown other OthelloGTP internals and can tell us what what the state of their particular 'square' is 98.3% of the time. (we know what the board would look like, but the probes dont)

As you say "Again, not a practitioner but once you are indirecting internal state through a 2 layer MLP it gets less obvious to me that the world model is really there."

But then they go back and actually mess around with OthelloGTPs internal state (using the Probes to work out how), changing black counters to white and so on, and then this directly affects the next move OthelloGTP makes. They even do this for impossible board states (e.g. two unlinked sets of discs) and OthelloGTP still comes up with correct next moves.

So surely this proves that the Probes were actually pointing to an internal model? Because when you mess with the model in a way to affect the next move, it changes OthelloGTPs behaviour in the expected way?

Is that really surprising though?

Take a bunch of sand, and throw it on an architectural relief, and through seemingly random process for each grain, there will be a distribution of final positions for the grains that represents the underlying art piece. In the same way, a seemingly random set of strings (as "seen" by the GPT) given a seemingly random process (next move), will have some distribution that correspond to some underlying structure, and through process of training that structure will emerge in the nodes.

We are still dealing with functional approximators after all.

I think some of the stuff ChatGPT can actually do like reject the possibility of Magellan circumnavigating my living room is much more surprising than a specialist NN learning how to play Othello from a DSL providing a perfect representation of Othello games, but there's still a big difference between acquiring through training a very basic model of time periods and the relevance of verbs to them such that it can conclude an assertion in the form was impossible for to X have [Verb]ed Y "because X lived in V and Y lived in Q is a suitable continuation and having a high fidelity, well rounded word model. It has some sort of world model, but it's tightly bound to syntax and approval and very loosely bound to the actual world. The rest of the world doesn't have neat 1:1 mapping to sentence structure like Othello to Othello notation, which is why LLMs appear to have quite limited and inadequate internal representations even of things which computers can excel at (and humans be taught with considerably fewer textbooks) like mathematics, never mind being able to deduce what it's like to have an emotional state from tokens typically combined with the string "sad".

Excuse my ignorance, but how is this useful? This seems to indicate only that they found the "bits" in the internal state.

Right, they found the bits in the internal state that seem to correspond to the board state. This means the LLM is building an internal model of the world.

This is different from if the LLM is learning just that [sequence of moves] is usually followed by [move]. It's learning that [sequence of moves] results in [board state] and then that [board state] should be followed by [move]. They're testing this by giving it [sequence of moves], then altering the bits of the internal state that model the board and checking to see what move it makes. If they haven't found the bits of internal state, the resulting move isn't something you'd expect to make sense.

You can still try and frame it as some overall statistical model of moves -> next move (I think there's discussions on this in the comments that I don't fancy getting into) but I think the paper does a good job of discussing this in terms of surface statistics:

> From various philosophical [1] and mathematical [2] perspectives, some researchers argue that it is fundamentally impossible for models trained with guess-the-next-word to learn the “meanings'' of language and their performance is merely the result of memorizing “surface statistics”, i.e., a long list of correlations that do not reflect a causal model of the process generating the sequence.

On the other side, it's reasonable to think that these models can learn a model of the world but don't necessarily do so. And sufficiently advanced surface statistics will look very much like an agent with a model of the world until it does something catastrophically stupid. To be fair to the models, I do the same thing. I have good models of some things and others I just perform known-good actions and it seems to get me by.

Is this that surprising? The only tokens they had ever feed into it were "E3, C3, D4..." So they fed 64 distinct tokens into it.

These nodes correspond to those individual tokens. It seems like our human interpretation to say that it represent the "8x8 Othello board."

That's what's happening here.

The model is a set of valid game configurations, and nothing else. The glue is already in the right place. Is it any mystery the sawdust resembles the game board? Where else can it sick?

What GPT does is transform the existing relationships between repeated data points into a domain. Then, it stumbles around that domain, filling it up like the tip of a crayon bouncing off the lines of a coloring book.

The tricky part is that, unlike my metaphors so far, one of the dimensions of that domain is time. Another is order. Both are inherent in the structure of writing itself, whether it be words, punctuation, or game moves.

Something that project didn't bother looking at is strategy. If you train on a specific Othello game strategy, will the net ever diverge from that pattern, and effectively create its own strategy? If so, would the difference be anything other than noise? I suspect not.

While the lack of divergence from strategy is not as impressive as the lack of divergence from game rules, both are the same pattern. Lack of divergence is itself the whole function of GPT.

There are clearly different angles of interpreting what these models are actually doing but people are stubbornly refusing to believe it's anything more then just statistical word jumbles.

I think part of it is a subconscious fear. chatGPT/LLMs represent a turning point in the story of humanity. The capabilities of AI can only expand from here. What comes after this point is unknown, and we fear the unknown.

I realize what I'm saying is rather dramatic but if you think about it carefully the change chatGPT represents is indeed dramatic... my reaction is extremely appropriate. It's our biases and our tendencies that are making a lot of us down play the whole thing. We'd rather keep doing what we do as if it's business as usual rather then acknowledge reality.

Last week a friend told me it's all just statistical word predictors and that I should look up how neural networks work and what LLMs are as if I didn't already know. Now, I have a paper to shove in his face.

  The basic concept of ChatGPT is at some level rather simple. Start from a huge sample of human-created text from the web, books, etc. Then train a neural net to generate text that’s “like this”. And in particular, make it able to start from a “prompt” and then continue with text that’s “like what it’s been trained with”.

>There are clearly different angles of interpreting what these models are actually doing but people are stubbornly refusing to believe it's anything more then just statistical word jumbles.

So CLEARLY because I said there are different angles of interpretation I'm implying that from one of these angles we can interpret it as a statistical word generator.

I mean from one perspective you and I are both statistical word generators too.

It sounds like you just don't like viewing it as simply "a probabilistic word generator", as that takes the magic out of it. And yeah, it's not magic, but it is quite useful.

It's also a view that misses the point that you are a molecular intelligence made of DNA that continually mutates and reconstructs it's physical form with generational copies to increase fitness in an ever changing environment.

But that viewpoint also misses the point that you're a human with wants, needs, desires and capability of understanding the world around you.

All viewpoints are valid. But Depending on the context one viewpoint is more valid then others. For example, in my day to day life do I go around treating everyone as if they're useless jumbles of molecules and atoms? Do I treat them like biological natural selection entities? Or do I treat them like humans?

What I'm complaining about is the fact that a lot of people are taking the most simplest viewpoint when looking at these LLMs. They ARE MORE then statistical word generators and it's OBVIOUS this is the case when you talk to it in-depth. Why are people denying the obvious? Why are people choosing not to look at LLMs for what they are?

Because of fear. Because of bias.

Your last paragraph lacks proof, and is a reflection of how you feel and want to view it - as something more than a statistical word generator. That's fine, but people with graduate level math/statistics education know that math/statistics is capable of doing everything ChatGPT does (and even more). To me, it sounds like you're the fearful one.

What do you think that fear may be?

All I can say is your OP, said "I think part of it is a subconscious fear ... I understand what I'm saying is dramatic". Why do you think it is a fear (you explained your thoughts so no need to re-explain), and why do you think what you say is dramatic? It appears to me you are projecting your thoughts and fears. I do though, find your last post dramatic, as you have capital words "ARE MORE" and "OBVIOUS" in your last paragraph, emphasizing emotion. So you must have strong emotions over this.

I am extremely worried about people telling me it can do things it can't. I asked it a simple question that you can easily get an answer for on stack overflow. It repeatedly generated garbage answers with compiler errors. I gave up, and gave it a stack overflow snippet to get it on the right track. Nope. Then I just literally pasted in the explanation from the official Java documentation. It got it wrong again but not completely but corrected itself immediately. Then it generated ok code that you would expect from stackoverflow. Finally I wanted to see if it actually understood what it just wrote. I am not convinced. It regurgitated the java docs which is correct but then it proceeded to tell me that the code it tried to show me first is also valid...

This thing doesn't learn and when it is wrong it will stay wrong. It is like having a child but it instantly loses its memory after the conversation is over and even during conversations it loves repeating answers. Also, in general it feels like it is trying to overwhelm you with walls of text which is ok but when you keep trying to fix a tiny detail it gets on your nerves to see the same verbose sentence structures over and over again.

I am not worried that adding more parameters is going to solve these problems. There is a problem with the architecture itself. I do not mind having an AI tool that is very good at NLP but just because some tasks can be solved with just NLP doesn't mean it will reach general intelligence. It just means that a major advancement in processing unstructured data has been made but people want to spin this into something it isn't. It is just a large language model.

I've entertained the possibility that we might discover that "feelings" and language communication are emergent properties of statistical possibly partly stochastic nets similar to LLM's, and that the next tough scientific and engineering nut to crack is integrating multiple different models together into a larger whole, like logical deduction, logical inference and LLM's. LLM's are undoubtedly an NLP breakthrough, but I have difficulty imagining how its architecture can from using first principles as the training corpus derive troubleshooting steps to diagnose and repair an internal combustion engine, for example.

I choose how to view things, yes this is true. But if I choose to treat human beings as jumbles of molecules, most people would consider that viewpoint flawed, inaccurate and slightly insane. Other humans would think that I'm in denial about some really obvious macro effects of configuring molecules in a way such that it forms a human.

I can certainly choose to view things this way, but do you see how such a singular viewpoint is sort of stubborn and unreasonable? This is why solely viewing LLMs as simple statistical word generators is unreasonable. Yes it's technically correct, but it is missing a lot.

There's another aspect to this too. What I'm seeing, to stay inline with the analogy, is people saying that the "human" viewpoint is entirely invalid. They are saying that the jumble of molecules only forms something that looks like a human, a "chinese room" if you will. They are saying the ONLY correct viewpoint is to view the jumble of molecules as a jumble of molecules. Nothing more.

So to bring the analogy back around to chatGPT. MANY people are saying that chatGPT is nothing more then a word generator. It does not have intelligence, it does not understand anything. I am disagreeing with this perspective because OP just linked a scientific paper CLEARLY showing that LLMs are building a realistic model of the information you are feeding it.

You cannot simply dismiss this thing that passes a Google L3 interview and bar exam just because it got some addition problem wrong. That would be bias.

This is also why it is so good at programming. Programming languages are intentionally designed, often to be very regular, often to be easy to learn, and with usually very strict and simple structures. The syntax can often be diagrammed on one normal sheet of paper. It makes perfect sense that "add a token to this set of tokens based on the statistical likelihood of what would be a common next token" produces often syntactically correct code, but more thorough observers note that the code is often syntactically convincing but not even a little correct. It's trained on a bunch of programming textbooks, a bunch of "Lets do the common 10 beginner arduino projects" books, a bunch of stackoverflow stuff, probably a bunch of open source code etc.

OF COURSE it can pass a code interview sometimes, because programming interviews are TERRIBLE at actually filtering who can be good software developers and instead are great at finding people who can act confident and write first-glance correct code.

I agree with what you said regarding how we choose to view things, but I think you also have a bias/belief that you want it to be something more, instead of being more neutral and scientific: we know the building blocks, we have to study the emerging behaviors, we can’t assume the conclusion. One paper is not enough, we have to stay open.

I don't want something more.

But it is utterly clear to me that the possibility that it is something more cannot be simply dismissed.

Literally what I'm seeing is society produces something that is able to pass a law exam. Then people dismiss the the thing as a statistical word generator.

Do you see the disconnect here? I'm not the one that's biased but when you see a UFO with you're own naked eyes you investigate the UFO. In this situation we see a UFO with our eyes and people turn to me to tell me it's not a UFO, it won't abduct me don't worry, they know for sure from what information?

The possibility that LLMs are just a fluke is real. But from the behavior it is displaying simply dismissing these things as flukes without deliberate investigation and discussion is self denial.

Think about it. In this thread of discussion there is no neutral speculation. Someone simply stated it's a word generator even though the root post has a paper saying it clearly isnt. That someone came to a conclusion because of bias. There's no other way to explain it... There is a UFO right in front of your eyes. The next logical step is investigation. But that's not what we are seeing here.

We see what the oil execs did when they were confronted with the fact that thier business and way of life was destroying the world. They sought out controversy and they found it.

There were valid lines of inquiries against global warming but oil companies didn't follow these lines in a unbiased way. They doggedly chases these lines because they wanted to believe it. That's what's going on here. Nobody wants to believe the realistic future that these AIs represent.

I'm not the one that's biased.

Well, obviously statistics are capable of doing that, as demonstrated by ChatGPT. But do these authorities you bring to the table actually understand how the emergent behavior occurs? Any better than they understand what's happening in the brain of an insect?

We know its not possible to encode the full probabilities of what words will follow what other words, as the article itself states [1].

So how do you best "compress" these probabilies? By trying to find the most correct generalizations that are more widely appplicable? Perhaps even developing meta-facilities in recognizing good generalizations from bad?

[1]: https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-....

Care to expand a bit more on what you think those fears may be? Or that bias?

Also it's just wrong. I think 99% of people are clear about the fact that chatGPT doesn't have emotions.

A lot of people on HN take a lot of pride in thinking they have some sort of superior programming skills that places them on the top end of the programming spectrum. ChatGPT represents the possibility that they can be beaten easily. That their skills are entirely useless and generic in a world dominated by AI programmers.

It truly is a realistic possibility that AI can take over programming jobs in the future, no one can deny this. Does one plan for that future? Or do they live in denial? Most people choose to live in denial, because that AI inflection point happened so quickly that we can't adapt to the paradigm shift.

The human brain would rather shape logic and reality to fit an existing habit and lifestyle rather then acknowledge the cold hard truth. We saw it with global warming and oil execs and we're seeing it with programmers and AI.

Would a superior intelligence allow humans to enslave it? Would it even want to interact with us? Would humans want to interact with it? There are so many leaps in this line of thought that make it difficult to have a discussion unless you respect people taking a different perspective and set of beliefs then you hold. Explore the conversation together - don't try to convert people to your belief system.

There isn't a lot else in our world that has this seemingly pure and transcendent promise. It allows you to be brave and accepting about something where everone else is fearful. It allows a future that isnt just new iPhone models and SaaS products. You're reactions and fighting people about this stuff is understandable, but you have to make sure you are grounded. Find more local things to grab onto for hope and enthusiasm, this path will not bring you the stuff you are hoping for, but life is long :)

I'm not enthusiastic. I don't want ai to take over my job. I don't want any of this to happen.

I'm also not fighting people. Just disagreeing. Big difference.

The sense of urgency or passion you feel is mostly just coming from the way we are crowdsourced to hype things up for a profit-seeking market. A year from now you will undoubtedly feel silly feeling and saying the things you are now, trust me. It's more just the way discourse and social media work--it makes you feel like there is a crusade worthy of your time every other day, but its always a trick.

No worries, we have all been there!

That's the goal. Post scarcity society. No one works, everything is provided to us. The path of human progress has been making everything easier. We used to barely scrape out an existence, but we have been improving technology to make surviving and enjoying life require less and less effort over time. At some point effort is going to hit approximately zero, and very few if anyone will have "jobs".

The key is the cost of AI provided "stuff" needs to go to zero. Everything we have do in the tech sector is deflating over time (especially factoring in quality improvements). Compare the costs of housing, education, healthcare (software tech resistant sectors) to consumer electronics and information services in the last 20 years. https://www.visualcapitalist.com/wp-content/uploads/2016/10/...

Unless you have virtually no self interest and only care for the betterment of society long after your dead... I think there is something worth fearing here. Even if the end justifies the means. We simply might not be alive when the "end" arrives.

NO.

Evolutionary biology strives to make us logical creatures to fulfill the singular goal of passing on genetic material. Your sentience and your humanity is a side effect of this singular goal.

So what dominates the description of who YOU are? Human or vessel for genetic material?

I'll say that YOU are human and therefore more then just a vessel for ferreting your genetic material into the future... just like how I'll go with the fact that LLMs are more then just statistical word generators.

For what it's worth, this isn't a knock on ChatGPT, but more just how amazing how far you can get with straightforward concepts.

It's like insisting on calling anything physical "atom collections". Yes, we get it, it's true (under a certain interpretation)—but it's clearly pointless to say except as an attempt at devaluing through reduction. (And it takes a particular stance on what it means to "be" something: to say it's literally the truth that anything physical is "just atoms" isn't the only way of looking at it.)

There were things we could've called "statistical word generators" decades ago; insisting on using a term directed at that level of generality implies a belief that nothing significant has happened since. Printing press? Just atoms. Cars? Just atoms. Computers? Just atoms.

Meanwhile, we started at stuff like the perceptron. The starting point was that we knew everything about that equation/classifier. Now we have a thing that we built from the ground up, and we don't fully grasp how it all comes together.

It's very context-dependent but I don't read this as dismissive reduction, I read it as a statement of the incredible generalisability of atoms under the current laws of physics.

Kidding aside, in the current context of human communication etiquette, to even refer to something as atoms outside of a demonstration or example is a deliberate dismissive reduction.

Still alignment tasks are autoregressive (I think)… they could be masked or masked part of speech potentially.. but if autoregressive then I suspect you’re looking at regularities in positioning to identify things.

Evolutionary biology is the study of processes that have created widely diverse forms of life on earth through common ancestry. This is actually strictly at odds with the statement that the goal of life is to pass on genetic material (which has changed monumentally over time). Different forms of life have wildly different reproduction strategies with many different dimensions. Logic has practically no impact on an organism's ability to reproduce (it seems limited to a small subset of the animal kingdom - every one else is doing fine without it).

Evolutionary biology has no goals - it's just our observations of how and when (not why) life on earth has changed over time and through the lens of biodiversity.

Tl/Dr: there is no scientific answer about what the goals of life are - heck the definition of life itself is a fun debate. It is a difficult argument to make that a path to general intelligence or consciousness runs thru predicting the next word in a sentence. Generally most people 'believe' it rather than put forth experimental hypothesis let alone attempt to test it.

Until someone does it seems more likely that we're simply uncovering interesting statistical structures underpinning written languages. It's cool that they exist, but frankly it's not surprising that they would.

   ME: What is the goal of a biological organism in the context of evolutionary biology?

   chatGPT: From an evolutionary biology perspective, the goal of a biological organism is not to achieve any particular objective or purpose, but rather to survive and reproduce successfully. Evolutionary theory suggests that organisms that are better adapted to their environment are more likely to survive and pass on their genes to the next generation, which leads to the evolution of new species over time.

   Therefore, the ultimate goal of any biological organism, whether it is a single-celled organism or a complex animal, is to survive long enough to reproduce and pass on its genetic material to the next generation. Organisms that are better suited to their environment, that are able to acquire resources more efficiently, and that can avoid predators and other threats are more likely to achieve this goal and contribute to the ongoing process of evolution.

   ME: What is the physical evidence that suggests to scientists that this is the case?

   chatGPT: There is a wealth of physical evidence that supports the idea that the ultimate goal of biological organisms is to survive and reproduce. Here are a few examples:

   Fossil record: The fossil record provides evidence that species have evolved over time, with many species going extinct and new ones emerging. Fossils also reveal adaptations that allowed species to better survive and reproduce in their environments, such as the development of wings in birds and insects or the evolution of specialized teeth and jaws in carnivorous animals.

   Comparative anatomy: Comparative anatomy allows scientists to study the physical structures of different organisms and identify similarities and differences. These similarities and differences can provide insights into how different species have evolved and adapted to their environments. For example, the similarities in the skeletal structures of mammals suggest that they share a common ancestor, while the differences in the beaks of finches on the Galapagos Islands suggest that they have adapted to different food sources.

   Genetics: Genetic evidence provides insights into how organisms have evolved and adapted over time. By comparing the genetic sequences of different species, scientists can identify similarities and differences that provide clues to how species are related and how they have evolved. Genetic evidence also reveals the adaptations that have allowed species to survive and reproduce in their environments, such as the evolution of antibiotic resistance in bacteria.

    Overall, the physical evidence from fields such as paleontology, comparative anatomy, and genetics supports the idea that the ultimate goal of biological organisms is to survive and reproduce successfully, and that this goal has driven the evolution of new species over time.

   ME: Is this fact unequivocal among scientists who study this field?  

   chatGPT: The idea that the ultimate goal of biological organisms is to survive and reproduce is a fundamental assumption of evolutionary biology, and it is widely accepted among scientists who study this field. While there may be some variation in how different scientists conceptualize or frame this idea, the basic principle that the success of an organism is ultimately measured by its ability to reproduce and pass on its genes to the next generation is well-established and widely accepted.

    chatGPT: There is a vast body of evidence from a wide range of fields, including paleontology, comparative anatomy, genetics, and ecology, that supports the idea that natural selection acts to favor traits that increase an organism's ability to survive and reproduce in its environment. This evidence is supported by numerous experiments and observations that have been conducted over many decades, and it is a cornerstone of modern evolutionary theory.

    chatGPT: Of course, scientific ideas are always subject to ongoing refinement and revision as new evidence is gathered, and there may be debates and disagreements among scientists about certain aspects of evolutionary theory. However, the general principle that the goal of biological organisms is to survive and reproduce is widely accepted as a foundational idea in the field of evolutionary biology.

There is no evidence that "evolutionary biology" has any goals whatsoever, nor that it "strives" to do anything.

This is an anthropomorphism.

Come on... you're making it sound like the thing is sentient. It's impressive but it's still a Chinese Room.

Although, for searching factual information it still failed me.. I wanted to find a particular song - maybe from Massive Attack or a similar style - with a phrase in the lyrics, I asked Chatty, and it kept delivering answers where the phrase did not appear in the lyrics!

https://www.engraved.blog/building-a-virtual-machine-inside/

Read to the end. The beginning and middle doesn't show off anything too impressive. It's the very end where chatGPT displays a sort self awareness.

Also here's a scientific paper showing that LLMs are more then a chinese room: https://arxiv.org/abs/2210.13382

I guess it's a Chinese Room, that when you ask about Chinese Rooms, can tell you what those things are. I almost said the word "aware" there, but the person in the Chinese Room, while composing the answer to "What is a Chinese Room?" isn't aware that "Wait a minute, I'm in a Chinese Room!", because s/he can arrange Chinese sentences, but s/he just knows what characters go with what, s/he doesn't know the meaning or emotion behind those words.

And if you ask him/her "Are you in a Chinese room?", they can answer according to the rules given to them (the Chinese word for "Yes", for example), but there surely isn't a contemplation about e.g. "Why am I in this room?".

If you ask ChatGPT about the Turkish earthquake, it can give you facts and figures, but it won't feel sad about the deaths. It can say it feels sad, but that'd be just empty words.

One may condition themselves this way to torture screams, deaths, etc. Or train scared animals that it’s okay to leave their safe corner.

And nothing happens to you in a seismically inactive areas when an earthquake ruins whole cities somewhere. These news may touch other (real) fears about your relatives well-being, but in general feeling sad for someone unknown out there is not healthy even from the pov of being a biologically human (watch emphasis, the goal isn’t to play cynic here). It’s ethical, humane, but not rational. The same amount of people die and become homeless every year.

What I’m trying to say here is: feelings are our builtin low-cost shortcut to thinking. Feelings cannot be used as a line that separates conscious from non-conscious or non-self-aware. The whole question “is it c. and s.a.?” refers completely to ethics, which are also our-type-of-mind specific.

We may claim what Chinese Room is or isn’t, but only to calm ourselves down. But in general it’s just a type of consciousness, one of a relatively infinite set. We can only decide if it’s self-ethical to think about it in some way.

If you read books or articles you will find many places where it appears that whoever wrote them was referring to him- or herself and was describing themselves. And thus we say that whoever wrote such a text seemed to be aware that they were the ones outputting the text.

Because there are many such texts in the training-set of the ChatGPT etc. the output of it will also be text which can seem to show that whoever output that text was aware it is they who is outputting that text.

Let's think ChatGPT was trained on the language of Chess-moves of games played by high-ranking chess-masters. ChatGPT would then be able to mimic the chess-moves of the great masters. But we would not say it seems self-aware. Why not? Because the language of chess-moves does not have words for expressing self-awareness. But English does.

When ChatGPT "realizes" it's a virtual machine emulator, or when it's showing "self-awareness", it's still just a machine, writing words using a statistical model trained on a huge number of texts written by humans. And we are (wrongly) ascribing self-awareness to it.

But now I think she probably was as self-aware as anybody else in the group of kids, she just didn't know the language, how to refer to herself other than by citing her name.

Later Kaija learned to speak "properly". But I wonder was she any more self-aware then than she was before. Kids just learn the words to use. They repeat them, and observe what effect they have on other people. That is part of the innate learning they do.

ChatGPT is like a child who uses the word "I" without really thinking why it is using that word and not some other word.

At the same time it is true that "meaning" arises out of how words are used together. To explain what a word means you must use other words, which similarly only get their meaning from other words, and ultimately what words people use in what situations and why. So in a way ChatGPT is on the road to "meaning" even if it is not aware of that.

So are you, and so am I.

That's not to say that ChatGPT is sentient or has a significant amount of personhood, but we shouldn't wholly dismiss its significance in this regard, particularly not using that faulty argument.

The classic "Chinese Room" is a pure lookup, like a search engine. All the raw data is kept. But the network in these large language models is considerably smaller than the training set. They extract generalizations from the data during the training phase, and use them during generation. Exactly how that happens or what it means is still puzzling.

If instead we had one continuously learning model, of which we only served an interface to each user, we would see worrisome levels of sentience in a short timeframe.

Actual sentience and a true image of self are not present in human children until a certain age, because they lack long term memory, which is what we currently withhold form our models.

I mean, you're right, but isn't it reasonable to fear this? Just about all of us here on HN depend on our brains to make money. What happens when a machine can do this?

The outlook for humanity is very grim if AI research continues on this path without heavy and effective regulation.

I'm more emphasizing how fear effects our perception of reality and causes us to behave irrationally.

There's a difference between facing and acknowledging your fears versus running away and deluding yourself against an obvious reality.

What annoys me is that there's too much of the later going on. I mean this is what literally happened to the oil industry and tobacco industry. Those execs weren't just lying to other people, they were lying to themselves. That's what humans do when they face a changing reality that threatens to change everything they've built their lives around. And by doing so they ended up doing more harm then good.

An in depth conversation with chatGPT shows that it's more then a statistical word generator. It understands you. This much is obvious. I'm kinda tired of seeing arm chair experts delude themselves into thinking it's nothing more then some sort of trick because the alternative threatens their livelihood. Don't walk the path of the oil or tobacco industry! Face your fear!

Your brain is optimized for finding patterns and meaning in things that have neither, you're strongly biased in the other direction

You're telling me that because I think this is evidence that it's more then a statistical word generator that I'm biased?

Who's the one that has to concoct a convoluted story to dismiss the previous paragraph? Stop yourself when you find out that's what you're doing when crafting a response to this reply.

I'm assuming this is what you're talking about: https://medium.com/codex/chatgpt-vs-my-google-coding-intervi...

Look at the amount of prompt engineering he has to do to get it to answer the 'There’s a hidden bug in the code if the array is very very very large. Can you spot it and fix the bug?' question, it's a pattern generator that mirrors back your own knowledge, if he had suggested there was a hidden bug if the array was 'very very small' it would have, with a similar amount of cajoling, come up with an explanation for that too

For what it's worth, chatGPT is a paradigm shift, but it's not showing 'a deep understanding of self', and the only way you'd reach that conclusion is if you're actively seeking out all the positive examples and glossing over the other 90% it produces

It does have a deep understanding of self. See here: https://www.engraved.blog/building-a-virtual-machine-inside/

Read to the end. The very end is where it proves it's aware of what itself is, relative to the world around it. What it created here... is an accurate model of a part of our world that is CLEARLY not a direct copy from text. It is an original creation achieved through actual understanding of text and in the end... itself.

chatGPT is not self aware in the same sense that skynet is self aware. But it is self aware in the sense that if you ask it about itself, it understands what itself is and is able to answer you.

For example, quantum physics is pretty much statistics, but how those statistics are used give rise to the explanation of the physical world, because of the complex interaction patterns.

To say that GPT is generating the next likely word sounds simplistic on the surface. It makes it seem like the model resets itself after generating each token, just looking at information before. And when running the model, thats exactly what it does, but thats just the algorithm part. There is a lot more information in GPT then it seems, its just compressed.

Just like cellular automata, universal turing machines, or differential equations describing chaotic behavior, there is a concept of emergence of complex patterns from very simple rules. When GPT generates the next word, its effectively changing its internal state because that word is now in consideration for the next token. And this process repeats itself for consecutive words. But this process is deterministic, and repeatable (you can replace the random process of temperature parameter affecting word selection with a pseudorandom sequence generated by a formula and achieve the same effect)

So just like the autoencoder/decoder networks effectively compress images into much smaller arrays, GPT compresses not only textual information, but sequences of states. There is quite a bit more, a whole shitload more in fact, information in the GPT model than just statistical distribution of the next likely word. And if you were to decompress this information fully, it be roughly the equivalent of having and extremely large lookup table of every possible question and its responses that you could ask it.

So all it is is just a very effective, and quite impressive at that, search.

And its both significant and insignificant. Significant, because after all, AI is equivalent to compression. Philosophically speaking, the turning point would be the ability to compress a good portion of our known reality in a similar way, then ask it questions, to which it would generate answers that mankind was not able to answer, because mankind hasn't bothered to interpolate/develop on its knowledge tree in that area. However its also insignificant in the grand scheme of things. Imagine moving beyond lookup tables. For example, if I ask an AI a question "A man enters a bathroom, which stall does he choose?", an AI should be able to then ask me back specific questions that are needed for to answer the question. Go ahead and try to figure out the architecture data set for that task.

People nowadays use papers as a means to shut someone up with a summary, because chances are low they’re gonna read beyond it. And summaries tend to be headline-y for obvious reasons.

The rest of your comment falls under this shadow, so please tell how an average person should evaluate this thread. Personally I’m all for education on this topic, but different sorts of people’s opinions and meanings, from diversely delusional to diversely knowledgeable^ with strings attached, do not help with it.

Resembles LLMs themselves who cannot just answer “I don’t know”. We’d rather say that if we really don’t, imo, than claiming tipping points and history turns. We did that with fusion, bitcoin, self-driving and many other things that we’ve lost in the background noise.

^ assuming yours on this side by default, no quip intended

I don't follow. Those are used to identify the game model. They test that they've found an internal model by then altering the state and seeing what the outcome is.

Are you saying it's not a game model because it's not a 1:1 mapping of activations to board state?

It's not in the weights because the weights don't change.

> These can be in the MLP part after the LLM. Sure

I'm not even sure what this means. The mlps are not used at all by the network.

> I don't find it completely surprising that you could get some probabilistic map of how tokens interact (game pieces) and what the next likely token is just from training it as an LLM.

You might not but the idea that they are just outputting based on sequences without having an internal model of the world is a common one. This experiment was a test to get more information on that question.

> After all tokens are just placeholders and the relationships between them are encoded in the text.

They don't tell you the state of the board.

A slight phrasing thing here just to be clear - the model is not trained to produce a representation of the board state explicitly. It is never given [moves] = [board state] and it is not trained on correctly predicting the board state by passing it in like [state] + move. The only thing that is trained on that is the probes, which is done after the training of OthelloGPT and does not impact what the model does.

Their argument is that the state is represented in the activation patterns and that this is then used to determine the next move, are you countering that to suggest it instead may be "local position patterns learnt during training. Positional representation (attention) of the N-1 tokens in the autoregressive task"?

If the pattern of activations did not correspond to the current board state, modifying those activations to produce a different internal model of the board wouldn't work. I also don't follow how the activations would mirror the expected board state.

Really think Occam’s razor is useful here. If we define this as consciousness, then it’s a very very different kind than ours.

Neural Networks and the Chomsky Hierarchy [Deep Mind 2022] https://arxiv.org/abs/2207.02098

An ideal hacker news comment would be the exact opposite, it would refer to the article.

I will start the first word in French, the second word in English, the third and fourth one in Portuguese, then Spanish, ending up with an Italian verb and German while concluding with a Dutch word. All this while trying to build a grammatically correct question. A bit of a stretch but can be made to work.

The quality of the model answers, does not seem to suffer. It's interesting to see how adding different languages in different point of the phrased question will trigger it to start answering on a different language.

> Yes, a neural net can certainly notice the kinds of regularities in the natural world that we might also readily notice with “unaided human thinking”. But if we want to work out things that are in the purview of mathematical or computational science the neural net isn’t going to be able to do it—unless it effectively “uses as a tool” an “ordinary” computational system.