redlock

I have quizzed it with three books (total more than 1500 pages) and it gave great answers.

Initially yes when they released 2 million context with Gemini 1.5 it wasn’t effective.

Try it with Gemini 3 pro/flash now.

This is a ridiculous statement. A simple example of the huge difference is context size.

GPT-4 was, what, 8K? Now we’re in the millions with good retention. And this is just context size, let alone reasoning, multimodality, etc.

https://x.com/polynoamial/status/1946478249187377206?s=46&t=...

Not really a good measure of quality or performance but of cost effectiveness

His quantitative easing and zero interest rate policy only saved asset managers and corporate balance sheets.

People today are worse off than before Covid even though the market is much higher.

Debt fueled growth (through quantitative easing and deficit spending) is not healthy and always has a bad ending.

Real economic growth is what was seen between 1950 and 1970 where purchasing power increased and most of the gains went to the middle class.

I am guessing he is referring to mechanistic interpretability research like these:

https://arxiv.org/abs/2405.16908

https://arxiv.org/abs/2407.03282

You are claiming they are statistical parrots, which I don’t think the parent poster meant.

The “statistical parrots” argument might have been compelling with GPT-3, but not with today’s models and the results of mechanistic interpretability research, which show internal representations and rudimentary world models.

Also, humans hallucinate/confabulate all the time. Llms even forget in the same way humans do (strong recall in the start and end of the text but weaker in the middle)

However, out of distribution generation and generalization are a better more useful metric. In these Yann Lecun has argued that interpolation is meaningless in the high dimensional spaces these "curves" are embedded in.(https://arxiv.org/abs/2110.09485)

ARC has proven they can generalize, and Alpha Go (not an llm but a deep network) has proven it can generate novel/creative solutions. We don't need AI to have a sense of self "I" for them to beat us at every human skill and activity. Infact it might be detrimental and not useful for us if AI developed a sense of self.

To me this proves that llms learn concepts and multilayeres representations within its network and not just some dumb statistical inference. Even famous llm skeptic like Francois Chollet doesn't invoke this stochastic parrots anymore and has moved on to arguing that they don't generalize well and are just memorizing.

With GPT-2 and 3 I was of the same opinion that they seemed like just sophisticated stochastic parrots, but current llms are a different class from early gpts. Now that o3 has beaten memory resisting benchmark like ARC-AGI I think we can confidently move on from this stochastic parrots notion.

(And before you argue that o3 is not an llm, here is an openai researcher stating that it is an llm https://x.com/__nmca__/status/1870170101091008860?s=46&t=eTe... )

And here is some experimental data

http://arxiv.org/pdf/1303.0614v1.pdf

A more extreme example of false beliefs causing great harm is the Holocaust that was caused by Nazis believing the Jews are part of a conspiracy to undermine Germany and were the cause of Germany's defeat in WW1.

False beliefs can cause a lot of harm to a lot of people in both personal level (not vaccinating your kids) and to the larger societal level.

It isn’t a coincidence that an Iraqi advised them. He saw the model in Kuwait.