News for hackers. Stuff that matters.
Re:A Skeptical Take on ChatGPT: Ezra Klein Interviews Gary M... by zwaps on Saturday January 07, 2023 @05:49AM (#34286959)
The main point here is that Large Language Models are crushing through barriers that people like Gary Marcus have previously deemed impossible to surpass. Check past episodes by these guys on Machine Learning Street Talk podcast (the whole "symbolic AI" crowd) for some gold quotes that just seem silly now!

I mean, we are not that far from 2017, when LLM's couldn't really write coherent text. And, for example, we had people saying then that they would never be really coherent because text is too sparse, see Chomsky etc. etc. Instead, all they would do is to copy+paste training examples. Naturally, these people didn't really understand how transformers form interdependencencies and how they are composing sequences in a much more complex fashion - as by now is obvious. However, at they time, and with their crude understanding of these models, these guys sure were confident to be right.

Next was Questions Answering, Reasoning, Math problems, Deduction, Coding... each things that LLMs could never do! They were so sure of it. And sure enough, a new model comes around that does it pretty well.

Researchers are bitter for two reasons.

First, We do not say that LLMs are perfect or general AI. Nobody says this. It's an obvious point not worth arguing over. But Gary Marcus et al. have become extremely popular. It's probably because some readers are slightly concerned about AI, and want someone to tell them that AI isn't really AI yet (duh) and won't replace their precious job (yet?). Their point is a strawman, and what goes beyond it (AI can't ever... X) is mostly wrong.

Second, however, these people have never contributed to the actual models. They are not part of the progress that - notably - is crushing through these barriers. They are bystanders. And it's freaking annoying, because their understanding of LLM's is imperfect (as is the case for everyone!), and yet they come up with these statements of absolute certainty. There is indeed research into these questions. But it is far, very far from resolved. In that sense, these people are - sorry to say - charlatans.

Well at least, everyone I know rolls their eye when the next Gary Marcus article or tweet comes around.

Re:A Skeptical Take on ChatGPT: Ezra Klein Interviews Gary M... by cfiggers on Saturday January 07, 2023 @07:26AM (#34287434)
> their understanding of LLM's is imperfect (as is the case for everyone!), and yet they come up with these statements of absolute certainty.

Oh interesting, so when they accuse ChatGPT of being dangerous because it can be "confidently inaccurate" it's the pot calling the kettle black

Re:A Skeptical Take on ChatGPT: Ezra Klein Interviews Gary M... by zwaps on Saturday January 07, 2023 @10:11AM (#34288677)
ChatGPT is trained via reinforcement learning to give answers that - ultimately - human non-experts would judge to be plausible. Answers were not trained to be correct or accurate - hence we should not expect ChatGPT to do well there.

Sociologists have found that humans seek plausible answers more than correct answers, especially when trying to make sense of a situation. For that reason, the training objective makes sense for a chatbot.

Now, anyone who tries to seel ChatGPT as giving truthful answers is of course doing a disfavor to the model and its training objective.

However, anyone also trying to sell the above either as

a) Deep insight, or

b) inherent limitation of LLMs

is indeed a pot calling the kettle black.

Going from plausibility to truthfulness is probably not a giant leap - at least not compared to what was achieved in the past three years. There's active research on it, and I am sure a good solution will arise.

However, ChatGPT is not that, and it isn't meant to be. It's not the training objective. Making this a huge point is again such a BS strawman as the other things I mentioned before.

Re:A Skeptical Take on ChatGPT: Ezra Klein Interviews Gary M... by mannykannot on Saturday January 07, 2023 @12:15PM (#34290167)
> ChatGPT is trained via reinforcement learning to give answers that - ultimately - human non-experts would judge to be plausible.

Ostensibly, it is being trained give replies that are more likely than most to be given by humans. On the reasonable (at least IMHO) assumption that people's speech is biased towards saying things that their target audience might see as plausible, this can be seen as doing what you say in the above quoted.

This is not, however, the same as being able to judge the plausibility of the semantic content of those statements, so I remain skeptical that the current methods for training LLMs are capable of creating that ability, which in turn leads me to be skeptical that going from plausible statements to reasonably reliably true statements will be a relatively small step in comparison*. I'm open to being surprised, however, as I already have been by what has been achieved so far.

The LLM-does-math paper you mention in your sibling comment looks extremely interesting, and maybe I will change my mind...

* At least by a continuation or scaling-up of current methods. Humans may be more likely to say truthful things than falsehoods (at least in some areas and some contexts) but a gap has formed between plausible and truthful LLM productions, and I don't see any particular reason to think more of the same will close it.

Re:A Skeptical Take on ChatGPT: Ezra Klein Interviews Gary M... by zwaps on Saturday January 07, 2023 @03:35PM (#34292466)
What do you mean by current methods though? ChatGPT is trained very differently method wise than GPT3 itself.

Whereas GPT3 was strongly based on sequential likelihood in a large corpus, ChatGPT is trained (i would claim) on producing outputs that are judged as good answer and hence plausible. Which is also why the recent iteration of GPT3 is still better than ChatGPT on some tasks.

Of course, a truthful model needs a new approach, and perhaps a new baseline model.

For instance, starting with code rather than text has been beneficial for recent GPT models, presumably by learning stronger reasoning compositionality.

Perhaps, some such baseline on truthfulness needs to be another starting point before large corpora language modeling.

Nevertheless, it seems to me this paradigm is more likely to be the way forward than any symbolic approach, say, which is not yet able to even produce text afaik.

I mean, even if we need a retrieval style fact checking component, I do now believe there’ll be a transformer equivalent LLM in any future model.

Re:A Skeptical Take on ChatGPT: Ezra Klein Interviews Gary M... by mannykannot on Saturday January 07, 2023 @08:53PM (#34295178)
Unless the difference in training of GPT-3 and ChatGPT has led to an improvement in the veracity of the replies given, I think the differences in their training are moot in regard to the question of what it will take to get substantially true responses from them. For the reasons I gave in my previous post, I don't regard progress in appearing plausible as necessarily progress towards being able to tell truth from falsehoods.

To be clear, I am not claiming that a symbolic approach would be better. Pointing out the shortcomings of current methods does not provide evidence that symbolic approaches will succeed (or vice-versa, for that matter.)

This leaves code. On reading your statement about starting with it, it suddenly occurred to me that code is a constrained environment compared to the whole of human discourse. It is obvious, I think, that programming languages are extremely limited in what they can express in comparison to human languages, but perhaps less obviously (at least to me), it seems to follow that what humans can say about programs is also constrained (again, in comparison to human discourse in general - but not nearly as constrained as what can be said in the programming languages themselves.)

This leads me to agree with you in this respect (at least tentatively), and here's why: Currently, on Earth, we have one species whose individuals have a well-developed sense of themselves as agents in that world, a similarly well-developed theory of mind about others, and are adroit language users. A few other species have some of these abilities to a limited extent, but not nearly so well developed that they can fully make use of them. As an evolutionist, I suppose that some of our ancestor and related species had intermediate levels of these capabilities, and if we can make AIs that occupy that space we would be making progress.

In this view, attempting to match or surpass human performance in language use and understanding the world is about as difficult a target as we could have. As an intermediate target, the domain of program code and the things people say about it has some things going for it: not only is it more constrained (as argued above), but quite a lot has been written about it and is available for training. It is also (arguably) less likely than human language in general to contain statements intended to influence opinions without regard to the truth.

Re:A Skeptical Take on ChatGPT: Ezra Klein Interviews Gary M... by elicksaur on Saturday January 07, 2023 @09:53AM (#34288484)
We have different definitions of pretty well. It can’t do basic math consistently.
Re:A Skeptical Take on ChatGPT: Ezra Klein Interviews Gary M... by zwaps on Saturday January 07, 2023 @10:23AM (#34288796)
If you prompt it correctly it seems to be able to solve most basic math problems you throw at it. Here's a PNAS claiming human-level performance: https://www.pnas.org/doi/full/10.1073/pnas.2123433119

There's a deluge of papers and research right now, so there's quite a bit of complexity to saying "pretty well".

However, let me say it this way: Compared to other LLMs, recent OpenAI models score highly on logic and math exercises. Yes, there ARE better LLM's trained to do math computations (especially fine-tuned for certain problems), but I'd say ChatGPT is certainly impressive as a general text and code model.

The other side of the medallion of saying "pretty well" is this: There is no other type of computational approach that is able to solve free-form logic or math queries in any capacity. There is no symbolic approach that can "extract" a math problem from text and then solve it - in code or otherwise, whereas LLMs are getting close to human performance on such (and related tasks).

So yeah. Pretty well is what I'd claim.

Re:A Skeptical Take on ChatGPT: Ezra Klein Interviews Gary M... by elicksaur on Saturday January 07, 2023 @11:49AM (#34289811)
As I understand that paper, I would take issue that the LLM isn’t “doing” math at all. Look at Figure 4 for the process to be most clear. It’s generating a text program that when run via Python can solve the problem. All the LLM is doing is matching the equation in the question to whatever operators it needs in Python syntax. I wouldn’t call that doing or understanding math.
Re:A Skeptical Take on ChatGPT: Ezra Klein Interviews Gary M... by zwaps on Saturday January 07, 2023 @03:23PM (#34292330)
So yes, you’ll probably want to take a model trained on math problems to forgoe the step with code.

But then, note that writing correct programs is indeed a high level solution to a full text math problem, is it not? Going from there to solving it directly should be a matter of some tuning.

Finally, who said understanding math?

The whole debate is about getting shockingly useful results precisely without symbolic reasoning.

If the main contention is that got does not do symbolic reasoning, then we are back at Gary Marcus… yes we know this. It’s not why researchers are so amazed by these models. Its that they output steps (or in this case code since its Codex based) solving university level math with a simple transformer architecture

Re:A Skeptical Take on ChatGPT: Ezra Klein Interviews Gary M... by gamegoblin on Saturday January 07, 2023 @10:54AM (#34289179)
It can't do math because it is operating in a single neural network path. You, also, cannot do math in a single neural network path. Even when you add two small numbers like 123+456 your brain is mentally iterating over the digits, detecting if a carry is needed, doing that, etc. That is, you have a looping/recursive process running inside your brain. You only output the final answer.

GPT does not have such a looping/recursive process inside its neural net. It's a fixed depth non-recursive neural net.

You can get it emulate recursive processes by prompting it with tricks like "think step by step". If you describe the addition algorithm you learn in elementary school (e.g. the digit-by-digit, carry if the sum exceeds 9, etc) in sufficient detail, it can execute that algorithm.

Re:A Skeptical Take on ChatGPT: Ezra Klein Interviews Gary M... by zwaps on Saturday January 07, 2023 @03:44PM (#34292561)
Gpt3 is a graph neural network with added recursive information through positioninal encoding. It outputs sequentially, but I am not sure why a RNN would be required beyond that.

I would agree that the manner of reasoning must differ, since the sequence follows bpe tokens rather than logical steps, however who is to say that another form of mathematical reasoning could not lead to valid results?

For instance, Gpt might solve the problem at each output step insofar as required to generate that token.

It certainly iterates over each output token, and the encoding of the problem is equivalent so iterating over the characters of the math problem, roughly speaking. But yes, the iterated output does not follow a logical graph externally, it is token by token. But internally, the network can absolutely follow a more complex graph.

Could you say what you mean by single network path when we speak about attention based architectures?

What sort of operation or information is missing in such architecture?

I am aware of of some results relating to certain graphs, but I do not think this would apply to a text describing a math problem, say.