News for hackers. Stuff that matters.
Re:Do Large Language Models learn world models or just surfa... by jacquesm on Sunday January 22, 2023 @08:59AM (#34477439)
It really is, on many levels. The thing that I'm mostly worried about is that bit by bit we will be moving critical decision making to systems that are poorly understood, be it self driving cars, insurance companies or banks deciding to let you borrow money - or not.

For now we have some degree of accountability, models need to be explainable for some small subset of the possible applications. But the overall mechanism seems to be that if there is a utilitarian perceived benefit that the solution is greenlit, without knowing whether this is just a statistical fluke which will steer us to longer term unintended consequences or whether it is a structural improvement. And once the decision is made to accept the model the ability to measure a baseline is destroyed so any 'creep' will likely go undetected. Such longer term consequences could be really damaging.

[dead]
Re:Do Large Language Models learn world models or just surfa... by chrismatic on Sunday January 22, 2023 @01:08PM (#34480312)
I guess they make up for it with accountability.
Re:Do Large Language Models learn world models or just surfa... by mindslight on Sunday January 22, 2023 @01:10PM (#34480324)
Yes... But humans generally have to explain ourselves. At the societal level, it's generally not good enough for another human to make an arbitrary decision. Rather they have to justify it with the reasoning leading to it. The better the reasoning, the more other humans believe them.

Whereas with regards to tech, society is currently at the level of "computer says foo" (so it must be true). Requiring explanatory output to justify decisions is a likely path forward for making LLM's accountable. But given how little we've been able to make traditional human-run tech companies accountable for things like sharing scoring formulas and abusing personal information, I'm not hopeful.

Re:Do Large Language Models learn world models or just surfa... by PeterisP on Sunday January 22, 2023 @03:54PM (#34482117)
It is well known with experimental evidence that when people justify their decisions with the reasoning leading to it, it does not necessarily have anything to do with the actual reasons for the decision; even ignoring lying, we do not have privileged insight into all the factors our brain used to make a decision, and when asked for a justification for any decision which wasn't 100% the result of explicit rational planning (which is almost every decision, as even high-planning decisions are demonstrably influenced by non-rational factors, intuition, etc), we use our social skills to effectively invent a post-factum reasoning that both sounds plausible and is socially acceptable - just as language models can do. We often deceive ourselves, we have biases that discourage us from admitting and acknowledging certain reasons for why we did or do something; in some cases extensive therapy can help us recognize why some justification or reason we told others and ourselves was a false one, etc.

In essence, dealing with plausible but potentially misleading justifications is something that we have had to do forever, and will still have to do for future artificial agents as well.

Re:Do Large Language Models learn world models or just surfa... by mindslight on Sunday January 22, 2023 @04:10PM (#34482271)
I fully agree [0]. And I can see how my original comment was worded such that you'd think I wasn't taking that into account ("the reasoning").

When using ChatGPT, I can't help but thinking "this sounds like a high school essay". But really, it's not. Rather, it's someone who has college+ level of reading and studying, but has never had to have their ideas tested or scrutinized. A user of ChatGPT kind of mitigates this by asking for clarification, which is a form of learning in the specific session. But in the real world, ChatGPT-as-student would be adjusted with that feedback and then incorporate it into its overall model for serving the next user.

[0] For example, the SSC post about "predictive processing" resonated strongly with me

Re:Do Large Language Models learn world models or just surfa... by jacquesm on Sunday January 22, 2023 @04:18PM (#34482338)
If they only add a feedback loop it will already improve hundredfold in a very short time with respect to the output because then suddenly people rather than endless unstructured input will guide the answers towards the expected outcome. That is definitely still a very far cry from actual intelligence but it may well become a lot more useful that way.
Re:Do Large Language Models learn world models or just surfa... by mannykannot on Sunday January 22, 2023 @06:06PM (#34483311)
I feel you are making a valid point here, but I have to demur where you say “…just as language models can do.” Current LLMs do not appear to have any concept of self, let alone a theory of mind - the notion that other people think of themselves the same way as one thinks about oneself (in fact, they don’t appear to have have any concept of language as being about an external world that exists independently of what is said about it.)

An LLM can sometimes be prompted to give a response that looks like an explanation for how it came up with a previous response, but it is important to realize that the actual process by which it came to make the original response bears no relationship to the “explanation” it gave. Because everyone's experience with language has been with human-generated language, which is often not completely rational but, when it is not, often deviates in ways we intuitively understand, it is difficult to see the responses of current LLMs as being fundamentally different, even when we know they are.

Re:Do Large Language Models learn world models or just surfa... by mansoon on Sunday January 22, 2023 @10:45PM (#34485228)
Your intuitive understanding is an illusion.
Re:Do Large Language Models learn world models or just surfa... by _benedict on Sunday January 22, 2023 @11:56PM (#34485638)
What you describe is quite different: humans may well mislead themselves or others when there is a reason to do so. But most of the time, if there is no cost or benefit associated with an explanation, a human is fairly good (with great variability in both person and context) at assessing why they did something or said something, or why they reached a conclusion - and they can get better at doing so with practice.

The fact the brain reaches decisions before we are consciously aware of them does not mean the brain does not tag decisioms with motives and other information necessary to explain them, though obviously we have to trust this information, and it may sometimes be wrong, but it has a pretty good track record and there’s no equivalent facility for AI today.

This holds especially true for the kinds of actions taken by AI today which are usually deliberated thought processes for humans.

This isn’t the same thing is saying all actions are explainable, but it is quite different to being a black box. In general, one only needs to question the explanations of a person if there is a good objective reason to do so.

> It is well known with experimental evidence that when people justify their decisions with the reasoning leading to it, it does not necessarily have anything to do with the actual reasons for the decision

Out of interest, what’s the highest quality evidence you are aware of in this area?

Re:Do Large Language Models learn world models or just surfa... by hoseja on Monday January 23, 2023 @07:54AM (#34488210)
Humans are:

- fundamentally limited

- rough edges sanded off in the ball tumble mill of evolution

Re:Do Large Language Models learn world models or just surfa... by klodolph on Sunday January 22, 2023 @01:53PM (#34480893)
Yes. I am worried that over the long term, models which are explainable or justifiable will be pushed aside by models which are nonexplainable and nonjustifiable, because the nonexplainable / nonjustifiable models are cheaper to get running and in the end we are all trapped inside a system that ruthlessly optimizes short-term costs.

I've only done a small amount of statistical work professionally, but I remember that it was basically a nightmare to try and collect the right data and use the right variables to get a useful inference. Other engineers on my team would try to improve the models by throwing more variables at them or increasing the number of parameters in the model and I was always pushing back on this. I felt a bit crazy pushing back on this--surely, using more variables and more parameters will make the model fit historical data more closely, and a model which fits historical data more closely will fit future data more closely, and a model which fits future data more closely will be more useful?

Explaining why this is wrong is really hard. I have deep respect for people who are able to explain the principles of statistics and statistical modeling.

Re:Do Large Language Models learn world models or just surfa... by thaumasiotes on Sunday January 22, 2023 @02:25PM (#34481308)
> I felt a bit crazy pushing back on this--surely, using more variables and more parameters will make the model fit historical data more closely, and a model which fits historical data more closely will fit future data more closely, and a model which fits future data more closely will be more useful?

> Explaining why this is wrong is really hard.

Seems straightforward. Points 1 and 3 are correct; point 2 is false.

Re:Do Large Language Models learn world models or just surfa... by klodolph on Monday January 23, 2023 @02:48AM (#34486465)
Points 1 and 2 are both wrong also.

Statistics is not intuitive.

Re:Do Large Language Models learn world models or just surfa... by thaumasiotes on Monday January 23, 2023 @05:49AM (#34487372)
Point 1 isn't wrong; it's correct. There's a reason the name for this problem is "overfitting".
Re:Do Large Language Models learn world models or just surfa... by klodolph on Monday January 23, 2023 @01:02PM (#34492486)
Yeah, it sounds like you’re assuming two things things here which are often (or usually) wrong:

1. No cross-validation, and

2. You have the ability to solve for additional parameters without losing accuracy.

Statistics is not intuitive.