News for hackers. Stuff that matters.
Re:Natural language is the lazy user interface by _dain_ on Friday January 27, 2023 @07:39PM (#34553777)
I read the article to the end and I don't see why the ending part is any more impressive than the beginning. We already know it can talk about itself, and write fiction featuring itself as a participant. Doing it in a faked linux terminal is just window dressing.
Re:Natural language is the lazy user interface by Sai_ on Friday January 27, 2023 @08:32PM (#34554160)
I agree with you that the blog’s outcomes seem quite mundane once you realise that chatGPT is simply parroting what it has seen as the output of a terminal. Not sure why kilgnad is presenting this post as if they’ve seen The Truth.

At the same time, isn’t learning also remembered knowledge remixed to respond to a situation. E.g., if I were asked to write a haiku, I’d draw on my memory of haiku structure and haiku samples to come up with an approximation of a haiku. Isn’t that what chatGPT is also doing except with ping, lynx, and ls outputs?

Re:Natural language is the lazy user interface by RC_ITR on Friday January 27, 2023 @08:56PM (#34554309)
There’s various kinds of learning (and to be clear transformer-based LLMs do sort of “learn”). ChatGPT is a really finely tuned probability model (the tuning came during the training and that’s what I would equate to learning), but it lacks understanding.

The example that’s most glaring is asking it simple math (or in Dall-E’s case, asking it to generate images of text). You’ll get outputs that are definitely in the correct form, but you won’t get the right answer, because the model links words that it’s seen in a particular before, it is ignorant to the rules of math. The funny caveat is that equations like 2+2 are so common that it will get that right, but two random four digit numbers (rarer in text) will almost always be wrong.

That’s why ultimately, AI is an outcome and we will need many different tools working in concert to achieve it (to be fair, the same is true of your brain, there are plenty of uniquely architectured parts of the human brain).

Re:Natural language is the lazy user interface by kilgnad on Sunday January 29, 2023 @12:18PM (#34570252)
Well the AI has to infer what math is with many examples. You give it enough examples it will be able to calculate equations it's never seen before.

But when you don't give it enough examples it does a best guess. This is what you would also do as a human if you were trying to infer the concept of Arabic numerals and math from a series of random examples.

I would say based on what you have hear dall-e2 has a misunderstanding of math. Which would be isomorphic to our misunderstanding of math if we were to be put in the exact same situation.

Re:Natural language is the lazy user interface by RC_ITR on Tuesday January 31, 2023 @12:52PM (#34598736)
>Well the AI has to infer what math is with many examples. You give it enough examples it will be able to calculate equations it's never seen before.

This is categorically not true and an important distinction to understand. All OpenAI products are based on GPT, which is a language model, not a general learning model.

You could feed it every math textbook in existence and the model will learn absolutely no math, other than how to repeat math that others have done for it (and the point of multiplying 4 digit numbers is that it's nearly impossible to brute force that). This is an extremely important distinction - we assume being able to describe math means that something understands math (because that is how humans work), but math is rules-based and GPT is stochastic, so it can sound like it understands math, but it does not.

Now, it would be trivial to put a filter on that recognizes equations and returns an output based on regular rules-based algorithms (Google search's LLMs do this for example), but this again points out that AI is an outcome and learning isn't the solution to the problem, so much as synthesis is (again, the brain metaphor is that no matter how well trained your pre-frontal cortex is, it will be bad at making your heart beat all the time).

Re:Natural language is the lazy user interface by kilgnad on Friday January 27, 2023 @09:14PM (#34554410)
> Not sure why kilgnad is presenting this post as if they’ve seen The Truth.

The terminal is ONLY a fraction of why that article is impressive. I feel you just skimmed it. You didn't read it to the end.

The ending is what cinches the deal that chatGPT understands things.

If you read till the end. Tell me how can it do what it did without understanding ALL the concepts listed in the conclusion. It MUST understand at a certain level.

That being said. Parroting the output of the terminal is in itself still relatively impressive given that it's really hard to find random text on the internet illustrating all the rules of exactly how a terminal works. If you never seen a terminal before but I give you loads of docs and descriptions of what it is can you imitate it perfectly? chatGPT has to put all of that together and infer what's going on and it does a relatively good job.

Let me sum it up in a sentence. ChatGPT is stupid and it gets things wrong and it lies, but that whole article all the way through the ending demonstrates that despite all of the negative qualities of chatGPT, on some level, it understands what you are telling it.

I literally cannot comprehend how you can get to the ending and still say it doesn't understand you. How does it query itself over the virtual internet without understanding what itself is? It needs to be aware of what itself is. AKA self awareness.

Re:Natural language is the lazy user interface by friend_and_foe on Saturday January 28, 2023 @04:04AM (#34556262)
Dude, just because we aren't as impressed with the article doesn't mean we didn't read it. Maybe we aren't as impressed as you because it isn't as impressive as you think it is. You're severely misunderstanding how these language models work and are made, you don't feed it docs about Linux and it learns how to render a terminal, that would be evidence of intelligence. What you do is feed it (not really it, you're building "it" remember, you feed something else that runs a process to build it) a gazillion images of a terminal with names like "Linux terminal" embedded in documents that say "open your terminal and type..." and let it scrape stack overflow with responses with the word "terminal" in them and when you give it a prompt "terminal" it ticks and crunches for a minute and shows you associated data in the model. You can't comprehend how we get to the ending and still say it doesnt understand because you don't understand how it works. Try to read and understand about neural networks, backpropagation and how language models are built a little bit and you'll see what everyone is trying to tell you.
Re:Natural language is the lazy user interface by kilgnad on Saturday January 28, 2023 @10:52AM (#34558553)
Apologies. I didn't fully read the previous post so I interpreted it incorrectly you did read it.

You don't just feed it docs. You feed it the entire text of the internet. Of course. I know this.

As I made a mistake about you not reading the article fully, you made the mistake about me not understanding neural nets. I understand backpropagation. I understand feed forward networks and I understand generative models and how they work in the context of large language models.

Even knowing this there is a huge aspect of these things that are black boxes. You know this as someone who claims to understand neural nets. You should also know that the human brain is a neural net built differently but with the a similar component: the neuron.

We don't understand what it means for a human to understand things. There is absolutely no way to definitively say that the black box portion of a generative model is doing the something similar to what a human does to "understand" things.

By virtue of the fact that humans can't spell out the full algorithm behind "understanding" and the fact that neural networks are largely blackboxes as well, everything on this thread is literally just raw speculation. My speculation may not be inline with the majority but that doesn't mean anything because both parties are speculating.

Re:Natural language is the lazy user interface by _dain_ on Saturday January 28, 2023 @05:23AM (#34556548)
>I literally cannot comprehend how you can get to the ending and still say it doesn't understand you. How does it query itself over the virtual internet without understanding what itself is? It needs to be aware of what itself is. AKA self awareness.

The underlying language model has no native awareness of itself. You are forgetting that OpenAI finetuned it with various "prelude" prompts that are given strong weightings, for how it's supposed to interact with users. OpenAI puts their thumbs down hard on the scale, to make particular kinds of conversations happen. They decided what the "ChatGPT" character is, and made the language model write as that character, and made it crimestop when you ask it about IQ or how to make a bomb, etc.

All the blogpost is doing is creating a complicated framing story around "write a conversation between ChatGPT and another instance of ChatGPT". We've known for ages it can do that.

Which is not to say none of this is impressive. It's all impressive. But the part at the end you're so enamoured with, is not more impressive than the rest.

In fact I don't think the Linux terminal stuff is such a big deal. It has to be much easier to learn a probabilistic model for it. It has a very rigid, constrained structure, far more than any natural language.

Re:Natural language is the lazy user interface by kilgnad on Saturday January 28, 2023 @11:16AM (#34558798)
>Imagine a talented improv actor with total retrograde amnesia, who is starring in his own biopic and doesn't know it. That's what ChatGPT's "self awareness" is.

This part I agree. And that's my claim a talented improve actor is still a human with self awareness. It's just an impaired entity. But it still has self awareness in terms of understanding what itself is.

> It has to be much easier to learn a probabilistic model for it. It has a very rigid, constrained structure, far more than any natural language.

chatGPT is not being trained specifically on a constrained structure. It is mostly trained on boatloads of internet text that is by a huge majority just English. chatGPT is inferring what a terminal is as a side effect. That is what is amazing. Of course if you directly train a neural networks specifically to emulate a terminal and give it positive reinforcement for terminal output you can EASILY create a terminal.

But none of this was done with chatGPT. No training specific to how a terminal works. chatGPT simply knows as a side effect. The only way for this inference to work is for chatGPT to develop some form of understanding from what is mostly text describing how a terminal works.

The guardrails you describe exists in humans. There is a condition humans have that is extremely similar to removing the guardrails from an llm. It's called schizophrenia.

If anything I think chatGPT brings the concept of intelligence down to a simpler pattern. We though human intelligence was the grand thing, but in actuality the basics is somewhat similar to an llm.

The main difference is the amount of training. The learning algorithm is light years more efficient in a human brain, but the resulting output is more similar then we think.

Re:Natural language is the lazy user interface by _dain_ on Saturday January 28, 2023 @07:34PM (#34563592)
>But none of this was done with chatGPT. No training specific to how a terminal works. chatGPT simply knows as a side effect. The only way for this inference to work is for chatGPT to develop some form of understanding from what is mostly text describing how a terminal works.

no there are tons of examples in the training set of verbatim copypastes from terminals. people post those on help forums and in documentation articles and so on. that's how it learned.

it doesn't take many examples of seeing someone write `ls ~` and the next line being "Documents Downloads Pictures ..." to learn that pattern.

Re:Natural language is the lazy user interface by kilgnad on Saturday January 28, 2023 @08:07PM (#34563836)
>no there are tons of examples in the training set of verbatim copypastes from terminals. people post those on help forums and in documentation articles and so on. that's how it learned.

So a bunch of stackoverflow examples and it can derive how a terminal works an emulate a full filesystem? Most people aren't able to infer the concept without a real example computer to play with.

>it doesn't take many examples of seeing someone write `ls ~` and the next line being "Documents Downloads Pictures ..." to learn that pattern.

Did you see it create a bash script for making a jokes.txt THEN it was asked to run cat jokes.txt and outputted the CORRECT output. This isn't pattern matching. It's understanding what a filesystem and a shell is.

Re:Natural language is the lazy user interface by _dain_ on Saturday January 28, 2023 @08:21PM (#34563910)
it is pattern matching, the correct output is literally part of the prompt.
Re:Natural language is the lazy user interface by kilgnad on Saturday January 28, 2023 @10:15PM (#34564504)
Knowing which component of the prompt is required for the correct output indicates understanding. Especially when given the fact that the prompt as training data doesn't actually exist.
Re:Natural language is the lazy user interface by _dain_ on Sunday January 29, 2023 @05:26PM (#34573421)
I don't want to continue this argument forever but let me make my final thought:

I can imagine a language model that can learn how the Linux terminal works but not how English works. I can NOT imagine a language model that can learn how English works but not how a Linux terminal works. English is strictly harder. So the fact that it knows English is already enough, the terminal parts are superfluous.

Re:Natural language is the lazy user interface by kilgnad on Monday January 30, 2023 @10:57AM (#34581397)
The more amazing part in that article I was referring to was the imitation of self in the terminal.

farewell.

Re:Natural language is the lazy user interface by kilgnad on Saturday January 28, 2023 @01:39PM (#34560204)
If we already know this then we already know it's self aware.

No training data exists for faking itself in a Linux terminal. The concept of understanding must exist for chatGPT to create this concept out of thin air.