Stochastic parrot is way too simplistic a dismissal. It's hand wavy.

But I don't get what's so impressive about the nuanced language of a language model that has been given datacenter amounts of compute and virtually all of written word ever put digitalized. Yeah it's the first actually functioning natural language interface. At what cost though. It's completely out of proportion with the benefits and only bubble level 'investing' can justify this.

Are you referring to the (one-time) training cost or the cost per inferred token? The latter is pretty acceptable these days, especially with smaller models.

Small models can't do impressive language corner cases. And the benefits of the models that can scale linearly while their cost grows geometrically.

Also claiming training costs are 'one off' when there's so much resources being poured into training new and bigger models is disingenuous.

The resources pouring in still pale in comparison to capital expenditures in the semi conductor industry.

The inflation in semi conductor capital expenditure is the problem caused by the AI hype.

Nah, tsmc spending tens of billion annually on capital before OpenAI.

> The theory goes - that if the model "understands" this scenario, then internally, it has has created something that models the real world.

But this is merely a definition of what it means to "understand" something.

For example just tabulating many input/output combinations would not follow this definition.

You're right of course. Let me rephrase. What I was trying to say was: If the model solves some reasoning tasks that may imply it's building some inner world model which is fundamental to reasoning. That in-turn might mean that the model is reasoning.

Others have argued against the "inner world model" theory and suggested that solving reasoning tasks is merely an extension of the "stochastic parrot" scenario - i.e., they claim that no such world model exists and that the model has rote memorized these reasoning scenarios.

Why not just give it a series of directions e.g. go one step forwards, turn right, etc like a LOGO program, and then ask it if it is back to the start or not. If the series of instructions is randomly generated then this is impossible to solve without a world model.

People have tried this! And for not-complex problems, the newer models do give the right answer and yet usually fail on the complex tasks.

Prompting does help though - if you ask the model to check it's own answer, it can often catch errors it made and improve. But the interesting part is precisely how does the model succeed at all at any spatial reasoning task? Surely it hasn't seen all spatial reasoning tasks in its training set. So, what's the other hypothesis? As you say, people suggest that there could be a world model that is constructed internally by the LLM.

However, as you can see right here in this thread that people disagree that you need a world model for solving such tasks.

From the preface to the 20th anniversary edition of Godel, Escher, Bach:

"Meaning cannot be kept out of formal systems when sufficiently complex isomorphisms arise. Meaning comes in despite one's best efforts to keep symbols meaningless! ...When a system of "meaningless" symbols has patterns in it that accurately track, or mirror, various phenomena in the world, then that tracking or mirroring imbues the symbols with some degree of meaning -- indeed, such tracking or mirroring is no less and no more than what meaning is. Depending on how complex and subtle and reliable the tracking is, different degrees of meaningfulness arise."

In other words, when one can reliably ask a language model a question and get a sensible answer, one is forced to conclude that it does in some sense "understand" what it is saying. This is also I think the essential philosophical thrust of the Turing Test, which is often misunderstood as a mere benchmark.

(I notice a common objection to examples of LLMs clearly demonstrating understanding is "it saw something similar in the training set". That may be true (though unfalsifiable) in any given instance, but the number of permutations of things LLMs correctly respond to far exceed the size of any training set. They are certainly generalizing, and interpreting their inputs on a conceptual level.)

It’s a type of understanding, but one we are not really familiar with, because it seems to understand tasks that it was trained on, but fails and other very basic ways on simple but different tasks.

But where do yo get access to these premium models? I only have free access to CoPilot (which claimed at one point to be GPT4) and it cannot solve any of these issues reliably.

It spectacularly fails at slight variations of the goat/cabbage/lion/river problem, it cannot solve spatial or mathematical questions reliably.

Do some of the researchers that hype up ChatGPT get access to a special version? I'm not inclined to buy a subscription to find out and the AI Reddits aren't that positive either.

How people get access to the premium models is typically to sign up[1] for an account and pay a subscription. GPT-4o is significantly different from original GPT-4 which is again very significantly different from GPT-3.5. This is all moving fast.

So if you don't want to pay for a subscription I think you can get some free use of anthropic's most capable model (Opus) - I don't know the status of what you can get for free from openai.

The opinion of AI reddit is only really going to get you so far because the capabilities of the models are wildly different for different use cases, so you really need to be able to try it out for yourself and see if it can do what you need it to do.

[1] Somewhere like https://openai.com/chatgpt/pricing/ https://claude.ai/ or similar

Like the sibling comment says, you typically have to pay for the premium model (and believe me, they are miles ahead of the free stuff - free copilot uses gpt3.5 which is a full 2 generations behind).

The other way is to get an API key from OpenAI or Anthropic and then, you can pay per token which is incredibly cheap - load it up with $5 and it can easily last you months if you're not using it much.

As a dev, I cannot recommend these better models enough. The ROI on my $10-$20 that I spend every month is easily in hundreds to thousands of dollars.