I hope nobody reading the above comment is swayed by it or thinks there is anything substantive there. If the implication is that if two things are the same from a "black box" perspective, how they actually work is just semantics, it's still false. Play with one of these models for a few seconds and it's obvious they are regurgitating and not thinking. If there was actually a chatbot that behaved like a sentient person, maybe the analogy would hold.

Furthermore, we know how a chatbot works, and it's not even pretending to simulate thought.

I'm not trying to downplay the language models, they are really cool, but ignorant people have mistaken them for something they are not (thought) and think that vague and profound sounding tidbits like the above post boost their case. If you're interested in these models, learn about how they are coded and trained, in the weeds, not from the podcast version that people like to parrot.

This seems like a very simplistic position on a rather subtle topic. I don't know of any good reasons to doubt that an arbitrarily large chatbot could pass an arbitrarily hard Turing test. If that's so, where is the bright line between Samantha and such a chatbot? The fact that current chatbots aren't very convincing shows that there's still a long way to go, but it doesn't demonstrate any categorical difference between chatbots and creatures that think - or that chatbots are categorically not "simulating thought" (whatever that means).

I mean, I'm certainly not here to argue that Samantha is definitely "thinking", for any definition of that term. But you seem to be suggesting that anyone who considers the idea must be ignorant or not understand how ML models are trained, and I don't follow that at all.

A lot of people have also mistaken humans writing plausible-sounding paragraphs for thought

We immediately start tripping over philosophical issues though. How is it possible to be confident that a language model is different from thought?

It seems quite possible that a model of thought could be equivalent to a language model and there is circumstantial evidence the two things are equivalent. A lot of people appear to think by reasoning something out with words. People are absolutely persuaded by words that pass a language model check, even if the words don't align to reality.

Personally I don't see how someone who understands how ML models work could think this way. It's a probabilistic model based on a corpus of text (actually arbitrary numbers representing tokens encountered in text) it's been trained on. It has not capacity to reason or to compare things with some understanding of how the world works, as people do. Any example to the contrary has been hard coded by a person. It's just a math equation, with billions of terms, to pick the best word that satisfies some criteria.

When we see ML models fail spectacularly, it's a reminder they don't understand anything, have no "common sense" they can test an idea against, unless a person explicitly wrote it.

I think it's hard to make a case to people who don't actually understand what an ML model is, which is why I mentioned learning about it in my previous comment. Nobody should believe what I (a random guy on the internet) say at face value, but it's a sophisticated science that's had a lot of unsophisticated folks jump on it and imagine things about it. Go to grad school if you really want to understand and make up your own mind.

> Personally I don't see how someone who understands how ML models work could think this way.

Then you've misunderstood the problem; because we all know how they work. What we don't know is how human reason works. We know from how lawyers and courts work that it is possible to reason about things purely using language from a corpus of text with no appreciation for how the details actually function.

A ML system reflects back the input it is given. If that input is human language, a very good ML system will produce output that looks like human language.

It is conceptually like a mirror. A good mirror, if you put a human in front of it, will produce output that looks exactly like a human. But we know it is not a human. Not because of some flaw in its representation, but because we understand how mirrors work.

But humans are also conceptually mirrors. For example, none of the words you are using are ones you thought up; they mostly date back to pre-Shakespearian times. Nor is the grammar you're using, which you've copied off other people. And the arguments you are using have concepts like intelligence or the mathematics of neural nets which aren't things that you came up with either; they are being mirrored off others.

And yet we'd all agree you are thinking, despite just being a mirror with very little room to add anything new. So if you want to argue that a neural net is a mirror, that does not in any way disqualify it from considering something. The part of human reasoning that isn't a mirror of other ideas is small to the point where it might not actually exist.

For GPT-3, I think an example makes it clearer. When GPT-3 was asked the question [1]:

> Explain the pun in the following joke: “What does Adult Swim call their physical retail stores? Brick and Morty.”

It answered:

> … The pun “Brick and Morty” alludes to the cable television network “Adult Swim”, which broadcasts a cartoon series called “Rick and Morty”, a humorous parody of “Back to the Future” and other science fiction films. “Brick and Morty” refers not to the characters in the cartoon, but to physical stores that sell DVDs and merchandise based on the cartoon. The pun conflates two meanings of “Brick and Mortar”, a brick-and-mortar store and a brick which is part of a building.

It seems on track until you get to the last sentence, which demonstrates that it does not understand the pun at all. There's other examples on that site where it falls down, like adding two 5-digit numbers.

Since it's able to answer well things that match the human-written text that it ingested, but falls down on simple things otherwise, that suggests that a lot what appears to be verbal understanding is actually from the (extremely large) amount of human-written source text it ingested, and not added by GPT-3 itself.

[1] https://www.gwern.net/GPT-3#pun-explanations

Is that a good example, though? The reason I was investigating pun explanations there was precisely because that was something I expected GPT-3 to not be able to do because of how its data is preprocessed to strip away phonetics. (If you've read that far in my page, you must've also read all my material ranting about BPEs.)

The puns & rhyming are striking in that they are exceptions that prove the rule about GPT-3's general linguistic capabilities, and are clearcut examples of how BPEs sabotage GPT-3: you can see how its performance falls off a cliff as soon as you hit a BPE-related task (like rhyming or alliteration or anagrams or dad jokes). It's one of the first things I look for these days whenever someone says GPT-3 doesn't do something it seems like it ought to be able to do: "is this phonetics-related, or could in any way be damaged by BPE encoding?" (And despite my warnings, people still get tripped up by it sometimes, or don't believe me - if I had a buck for everyone in the OA Slack or forums who'd tried to make GPT-3 rhyme to prove me wrong...)

But I don't believe there is anything intrinsic to Transformers or self-supervised language modeling which requires this. BPEs are just a cheap performance hack that everyone does without thinking too much about. I expect that switching to character-level (or at least phonetics-aware) encodings and spending some more compute/data would fix all of the issues I diagnose.

You're right, it's not a good example. I either forgot the context or didn't read it thoroughly enough when it was first written.

I should get more familiar with the current state of AI, I've never been a long-run skeptic but my projects are unrelated so I probably have a poor idea of how close we are.

Humans may even "be conceptually mirrors" (whatever that may mean): this still doesn't make mirrors be humans (conceptually or otherwise).

Current language models are much more closer to "Dissociated Press" (i.e. an equation that tells you which which words are more likely, given the context) than to "human thought", and the fact that humans learn by copying does not really change this.

But it does mean mirrors might be humans/human-like, if humans are actually reducable to (self-aware) mirrors.

Would you say that a "Markov chain"-type (e.g. Dissociated Press) language model is "self-aware" in any way?

If yes, then... how, exactly? It is basically an N x N matrix of values.

Current language models (GPT et al.), are qualitatively nothing fancier than this: probabilistic models which encode regularity in language and from which you can sample to get "plausible content".

If a bunch of values in a matrix is "self-aware", then I guess GPT can be seen as "self-aware"; if not, then it can't.

My problem is trying to imagine an N x N matrix being self-aware (like... what does that even mean in this context?).

Is GPT human-like? Sure... if you stretch the meaning of "human-like" enough (it produces content that is similar to content produced by a human). Is a human GPT-like? That's harder to argue (and I don't see how your argument would support it).

If you know what a Markov chain is then you must also know that modern language models are nothing like Markov chains. Just as an example, a Markov chain can't do causal reasoning or correctly solve unseen programming puzzles, the way GPT-3 can.

As for self-awareness, your brain is an N x N -matrix in the same sense as an ANN, so surely it must be possible for one to be self-aware? Not claiming that GPT-3 is, of course.

> If you know what a Markov chain is then you must also know that modern language models are nothing like Markov chains. Just as an example, a Markov chain can't do causal reasoning or correctly solve unseen programming puzzles, the way GPT-3 can.

First, GPT-3 can hardly do "causal reasoning" beyond exploiting regularities in the content it has been trained on. That's why you get "interesting causal reasonings" such as "the word 'muslim' co-occurs a lot with 'terrorist', so these things must be causally related".

Second, just because a more complicated version of a Markov chain (i.e. a probabilistic language model) can do things that a first-order Markov chain cannot does not mean that it is qualitatively different: both things are nothing more than simple mathematical models (linear algebra + sprinkles). A polynomial model can do things than a linear model cannot, but it is no closer to consciousness and self-awareness, as far as I can tell.

My point is... a mathematical model is just that (a model) and, as such, cannot be "self-aware".

Humans, as cybernetic agents embedded in some environment (from which they get sensory input and with which they can interact), have agency and, as such, can display the property of "self-awareness". Models, by themselves, cannot.

Humans may contain systems within them that resemble a GPT-3 language model (or a Markov chain model), and such "language models" may even be required for self-awareness (it's not obvious, but let's assume it's true).

*Still*, it is not the language model itself that is self-aware, but the agent that is using the model.

A GPT-3 model, by itself, cannot be self-aware, because it not much more than a applying some linear algebra operations on numbers (just like e.g. a Markov chain language model). An agent containing (among many other things) something that could be approximated by a GPT-3 model within, may.

> As for self-awareness, your brain is an N x N -matrix in the same sense as an ANN, so surely it must be possible for one to be self-aware? Not claiming that GPT-3 is, of course.

No, it is not. My brain is an analog computation device and not a bunch of numbers. Perhaps you can approximate some aspects of how it works using numbers, probably using digital computation devices, but that is not what anyone's brain is (neither at the "hardware" level, nor at the "software" level). Also, notice that a brain always exists within a biological agent, and it is the biological agent that is (or may be) self-aware, and not "the brain".

Sorry that my answer comes so late, but I'll put this here for posterity. I will only address the latter part. My point was that a brain is an N x N -matrix in the same sense as an ANN. An ANN is no more an N x N -matrix "in reality" than a biological brain; in reality it is some collection of analog electric potentials and configurations of matter which can sometimes be represented as a digital N x N -matrix for the convenience of the programmer. Thus the situation is exactly identical to a biological brain, which is also not "in reality" an N x N -matrix but can be represented as one. If we had sufficiently advanced (nano-)technology, we could manipulate human brains through their abstract representation as a matrix just as we can ANNs. Any distinction is purely pragmatic.

In any case I was not saying that being representable as an N x N -matrix is sufficient for consciousness (which I do not believe), simply that it is clearly compatible with consciousness. I agree that a self-aware ANN would probably require a body (possibly simulated) and some notion of agency.

Humans are net positive; that’s where all those words and grammar and ideas came from. As a practical example, Google says it sees about 15% new queries every year (queries that have never before been entered into Google).

Is that a good proxy for how net positive people are in general? I don’t know, and the exact number may not be stable or even really matter.

Human intelligence is kind of a red herring in this discussion, because our intelligence rides on something deeper, something we share with all animal life. Call it “will,” or “drive,” or even “soul,” the name doesn’t really matter. What matters is that we make decisions with that deeper thing, and then we apply our intelligence to execute and interpret those decisions.

What’s your favorite food? Who are you attracted to? What sounds like a fun thing to do today? Why do you sometimes feel like not going to work? We can think about these answers, and how to get what we want, but we can’t really logically reason our way into changing what we want. Not all of them, anyway.

That’s what is missing from ML systems, and why they are clearly tools. If you train up an ML system and then leave it alone, it just sits there until you prompt it. If you raise a human, they do NOT just sit there until you ask them to do something. And there’s a good chance they might not do what you want even if you ask, because they have “wants” as well.

To go back to my mirror analogy, people don’t suddenly step out of mirrors and disagree with us. And ML systems don’t suddenly want to write a novel instead of respond to whatever question we input. They’re not alive, they’re just really good at mirroring back written language when we give them written language as an input.

Edit to clarify: I’m not claiming that “will” has to be metaphysical or supernatural. Even if you believe humans are purely physical systems (as I do), we can characterize their behavior, compare it to ML systems, and see what’s missing.

And more to the point, we know it is missing from ML systems since we designed those systems and we did not put it in there. We should not ignore our existing knowledge of how an ML system works when thinking about its output. Just like we shouldn’t ignore our knowledge of how mirrors work when looking at a perfect reflection of a person.

I don't think that analogy holds. It would work if I'd said "It's not talking, it's just vocalizing a series of words that resolve into understandable sentences". Which would, of course, be 'talking'.

But I said it wasn't 'considering', which it isn't. It doesn't know things; it has no internal experience. It's not capable of understanding a concept, or really understanding anything at all. It's not programmed to do those things, nobody informed about the subject expects it to do those things, there's no place in the model for any of those things to hide if they've somehow arisen spontaneously.

Note that this is not some sort of luddite "a machine can never be alive, AGI is impossible" sort of rant; I think it's perfectly possible that someday we'll build machines that know and think and feel and are alive in every sense that a person is, maybe more so. But GPT-2 or GPT-3 and machine learning in general aren't that - they are the first tentative steps towards that future.

If we're building a brain, machine learning is a couple of hundred neurons in a petri dish somewhere that we've coaxed into connecting together in novel ways and displaying emergent behavior. It's a big deal, it's important, but it's not thinking for itself yet.

I don't really follow. What does it specifically mean to say that a chatbot has no internal experience, or can't understand a concept? Like, if I disagree with you about those claims, what observable things about the chatbot show that I'm wrong?

Or look at it another way - the posts from Samantha in TFA are just words on a page, same as the ones you wrote and the ones I'm writing now. Sure, some guy claims Samantha's words came from a chatbot - but suppose we don't believe him. Does something about the words themselves show that they came from an agent with "no internal experience"? If not, then what does it mean to dismiss the words on that basis?

> if I disagree with you about those claims, what observable things about the chatbot show that I'm wrong?

The most cursory Turring test

A Turing* test is a black-box test that only considers the subject's output. It specifically doesn't try to answer questions about how the subject works internally, like GP asked. That's the whole point of the thing!

It can't prove internal states, but it sure can rule them out.

Only contingent on a good faith assumption.

An AI intelligent enough to pass the test may have the internal state to decide to fail the test for inscrutable reasons, and if it's intelligent enough it will successfully fail.

It's not even hard to imagine how this could happen. For example an AI that evaluated that itself would be firewalled from the environment by worried humans if they found out it's more intelligent than the humans are comfortable with. (Perhaps this comment will be part of an AI's data corpus, seeding the idea. I'm probably not saying anything new, though.)

Thus the Turing test can't rule out internal states.

Counterpoint: the question is meaningless.

A "Turing test" is basically a notional thought experiment - useful to reason about, but in actual practice it's nothing more than a person reading some text and making guesses about who wrote it. And the "internal states" we're talking about here - "having internal experience", "understanding a concept" - are vague labels with no concrete definitions.

So in this context, asking whether "Turing tests can determine internal state" boils down to asking whether a person can subjectively determine something subjective.

In my take on this, it doesn't matter if "rule out internal states" is meaningful, or just a string of words used as vague labels, uttered by a subjective examiner with no criteria in common with anyone else.

If the examiner has in their belief reportoire "I have ruled out internal states!" (as well as the negation of that), regardless of however flawed that idea may be, just a series of vague labels with no concrete definitions, I would argue that a sufficiently intelligent AI may still trick the examiner into not adopting that belief.

Therefore, relative to the meaning held by the examiner of "internal states", the AI may objectively prevent the examiner from "ruling them out", whatever that means to the examiner. The AI's capability works out the same in either case, and more interestingly, anywhere else along the spectrum from meaningless to meaningful.

This relative kind of reasoning (i.e. not needing to define all the terms, even allowing that they aren't necessarily meaningful) is practically quite useful in many fields. In the case of the AI that decides to hide it's apparent inner states according to the criteria of the examiner due to reasoning about the consequences of the examiner's assessment, that's a practical consequence, despite the examiner's question being potentially meaningless.

I follow you, but disagree. If P is meaningless, we can't say anything useful about it - even "!(P&!P)" doesn't necessarily hold!

As for your specific argument, it begs the question. If you assume that a sufficiently smart AI could always convince someone it lacks P, then you've already assumed the premise (a sufficiently smart person can detect P) to be false. Of course you may find the assumption reasonable, based on your intuition about what P means, but someone who disagrees would just reject the assumption.

This is all the wrong attitude and we should be careful. I'm writing yet another market trading bot, and this one implements the fear and greed index (https://money.cnn.com/data/fear-and-greed/).

I explained to a non-technical friend that I was implementing "fear and greed" in my bot as a kind of joke, and he said, "Oh wow you're giving it emotions. It's sentient? That's really cruel to make it fearful. Why would you do that?"

I tried to spend the next ten minutes explaining it was just a series of "if" statements. I'm not sure he really got it -- all he said after was something like "It's amazing programmers can do that now"

lol by that logic we should be careful in how we trade because we wouldn't want to make "the market" fearful. Also, that link interesting because I've only ever followed the VIX and didn't know about the others in that list.

True, but telling people a program is alive or sentient when it's not is spreading a false hood.

Nice, and I hadn't used the VIX before. We both learned something today. :)

> telling people a program is alive or sentient when it's not is spreading a false hood.

If you think my comment was doing that, I think you may be misremembering the Dijkstra quote.

Ah, I'm enlightened on another level, having looked up the quote now. Thank you -- that's TWO things I've learned today!

Does a submarine swim?