Building a ChatGPT Enhanced REPL

Re:Building a ChatGPT Enhanced REPL by OrderlyTiamat on Saturday April 22, 2023 @03:05AM (#35664084)

You are right on fhe first point, I believe the second needs to be corrected. LLMs have no state. Each word- each token- is made anew, the network being oblivious to the process that led to the previous token. It cannot even distinguish between text it generated and text you added unless you tell it that. So any relationship between output and explanation is 'coincidental'. Actually, any relationship between two tokens next to each other is 'coincidental' in the exact same way, so it cannot explain its reasoning. If you prompt it to comply or explain why not, the network may generate an explanation of why it can't comply, but it is doing that simply as a good continuation of the input text.

I'm not saying you can't call that an explanation if you want to use that word, I'm trying to make a distinction between the generated output and an explanation of that output. All of this is the former. The latter is not possible unless you analyse the model as it is running, and we don't really have a good understanding of that process yet.

There was a cool paper where researchers analysed a finetuned model playing othello, and they found some neurons that clearly represented the game state- if they changed those neuron values, the network responded correctly, even if that game position was impossible to get to normally [1]

I'd argue work like that is getting us closer to getting explanations for llm behaviour.

[1] https://www.joyk.com/dig/detail/1674380676520322

[ Parent | on HN ]

Re:Building a ChatGPT Enhanced REPL by og_kalu on Monday April 24, 2023 @02:31PM (#35691321)

We don't know if there's a state or not. There probably is some kind of state. Many kinds of generations are impossible without it.

>It cannot even distinguish between text it generated and text you added unless you tell it that.

Many people cannot distinguish reality from fiction invented by the brain. This doesn't mean much. If i could insert visual sense into you anytime i wanted, you'd struggle to tell it apart at best.

People can't really recreate mental states. It's all post hoc rationalization. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/

and the split brain experiments tell us the brain will happily fabricate those rationalizations to be something entirely unrelated. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/

[ Parent | on HN ]

Re:Building a ChatGPT Enhanced REPL by OrderlyTiamat on Tuesday April 25, 2023 @02:50AM (#35697297)

I don't know what this all means. You'll have to explain your point about fabrication as it relates to state in llms in more detail.

But as a matter of technical implementation I can tell you: there is no state. You can't simply say "we don't know if theres state" and be done with it. You individually might be unsure. Check the paper "attention is all you need", which introduced transformers, if you'd like to see for yourself [1].

The state is all in the text, that is why I said it's important that llms can't even distinguish between text you wrote and it wrote- how can you talk about an internal state of the llm if I can create that state by my input of text? That's just input at that point, not state irrespective of input. The same would go for your inserted visual sense, I suppose.

[1] https://arxiv.org/abs/1706.03762

[ Parent | on HN ]