News for hackers. Stuff that matters.
Re:Hot take: GPT 4.5 is a nothing burger by JohnKemeny on Saturday March 01, 2025 @01:49AM (#43216641)
I, for one, agree with you, and my area is cognitive science and artificial intelligence. It's staggering to see the hive-mind of Sam Altman followers chanting about AGI in a Markov chain.

Yes, we can today make natural-sounding hyper-average text, images, and perhaps even music. But that's not what writing and creating is about.

Re:Hot take: GPT 4.5 is a nothing burger by kranke155 on Saturday March 01, 2025 @03:26AM (#43217250)
As a writer, people thinking about LLms writing a book-length story don’t realise it’s in the wrong problem space.

LLMs can solve equations and write code, like make a React web page.

So they assume “a story is just an equation, we just need to get the LLM to write an equation”. Wrong imo.

Storytelling is above language in the problem space. Storytelling can be done in multiple methods - visual (images), auditory (music), characters (letters and language) and gestural (using your body, such as a clown or a mime, Im thinking of Chaplin or Jacques Tati). That means storytelling exists above language, not as a function of it. If you agree with the Noah Harari vision of things, which I do, then storytelling is actually deeply connected somehow to our level of awareness and it seems to emerge with consciousness.

Which means that thinking that an LLM that can understand story because it understands language is… foolish.

Storytelling lives in a problem space above language. Thinking you can get an LLM to tell a story because you can get it to write a sentence is a misunderstanding of what problem space stories are in.

It’s the same as thinking that if an LLM can write a React web app it can invent brand new and groundbreaking computer science.

It can’t. Not for now anyway.

Re:Hot take: GPT 4.5 is a nothing burger by namaria on Saturday March 01, 2025 @03:31AM (#43217277)
I agree. Language manipulation is a projection of something higher order. LLMs can shuffle the projection but they clearly cannot go over the gap to the higher order thinking.
Re:Hot take: GPT 4.5 is a nothing burger by fooker on Saturday March 01, 2025 @04:06PM (#43223679)
> AGI in a Markov chain

Do you understand why modern LLMs are different from Markov chains?

Re:Hot take: GPT 4.5 is a nothing burger by echoangle on Saturday March 01, 2025 @05:50PM (#43224723)
I actually don’t, can you explain it?

Doesn’t the AI model basically take an input of context tokens and return a list of probabilities for the next token, which is then chosen randomly, weighted by the probability? Isn’t that exactly the definition of a Markov chain?

Re:Hot take: GPT 4.5 is a nothing burger by fooker on Saturday March 01, 2025 @07:08PM (#43225587)
You have almost answered the question.

LLMs basically return a Markov chain every single time. Think of it as a function returning a value vs returning a function.

Now, I'm sure a sufficiently large Markov chain can simulate an LLM but the exponentials involved here would make the number of atoms in the universe a small number. The mechanism that compresses this down into a manageable size is famously 'attention is a you need.!'

Re:Hot take: GPT 4.5 is a nothing burger by JohnKemeny on Sunday March 02, 2025 @06:18AM (#43229314)
> Now, I'm sure a sufficiently large Markov chain can simulate an LLM but the exponentials involved here would make the number of atoms in the universe a small number.

No, LLMs are a Markov chain. Our brain, and other mammalian brains, have feedback, strange loops, that a Markov chain doesn't. In order to reach reasoning, we need to have some loops. In that way, RNNs where much more on the right track towards achieving intelligence than the current architecture.

Re:Hot take: GPT 4.5 is a nothing burger by fooker on Sunday March 02, 2025 @07:34PM (#43236956)
A modern reasoning model is exactly a feedback loop.
Re:Hot take: GPT 4.5 is a nothing burger by seanhunter on Sunday March 02, 2025 @10:53AM (#43231653)
They have the Markov property that the next state depends only on the current state (context window plus the weights of the model) do they not? Any stochastic process which possesses that property is a Markov process/chain I believe.
Re:Hot take: GPT 4.5 is a nothing burger by fooker on Sunday March 02, 2025 @07:35PM (#43236972)
The universe has that property too.

But no, most LLMs have a tweakae 'temperature' parameter that introduces some randomness and sometimes have very interesting results.

Re:Hot take: GPT 4.5 is a nothing burger by seanhunter on Sunday March 02, 2025 @11:36PM (#43238346)
Temperature just changes how an llm samples from the token distribution so I wouldn’t think that constitutes “memory” that would contradict the Markov property