I didn’t plagiarize! I merely have deep appreciation for the syntax and language idioms of the New York Times.

I don't think they are saying its regurgitation, but rather that the "approaches and patterns" used to build a python interpreter are in the training data because the training data asks the llm to build one.

>All it got from that was an appreciation for syntax and language idiom;

Sounds like it's you who is behind the times. With things like REA we know that LLMs can reverse engineer the behaviour of binaries and basically treat them as a form of specification to rebuild an application.

This means that old python interpreter acts like a specification for what kind of features must be included inside the project.

LLMs are amazing at input to output transformations and that hasn't changed at all.

I think it's you who has to update his mental model away from "stochastic parrot era" accusations.

PS: if you're trying to accuse me of being a "stochastic parroter", that's exactly what I mean.

Taking an input and transforming it to an output is the exact opposite of repeating something.

Well then ‘it’s in the dataset’ doesn’t matter. An agent can just pull down the code as part of its research when it’s tasked with a similar rewrite. That seems far more useful.

People regularly write here that nobody really knows how LLM weights are working really, we only know how to achieve that and run inference. Isn't that true?

I see it this way in layman sense:

- you feed the model with data;

- you run the query;

- you activate a network and it gives you most probable token, refeeds it back and generates next and so on;

- nobody knows exactly why and how this next token comes and can tell strictly which input data gives which output data without running the exact query with 0 temperature.

So yes, it's not an exact copy, but you also can't say that a particular repo is not reflected anyhow. And I mean that the weights are reflected by the particular logic, not a syntax.

Otherwise why all the hassle if we can just feed the language reference doc to 100mb model and it should be able to produce everything just fine if only the syntax connections are needed and everything else is freshly derivated and not based on the actual implementations.

I was speaking of reinforcement learning with verifiable rewards. It's not clear to me from your comment why re-writing python couldn't be in the set of training tasks used during RLVR.

Then maybe phrase it less dismissively?

‘It’s in the dataset’ sort of implies ‘the model is just reproducing output it was trained on’.

In this case you’re saying ‘the model is repeating tasks it was trained to do’

That’s the difference between accusing someone of just regurgitating something they read, vs praising them for having prepared well for an exam.

This doesn't really make sense. If you have the specific questions of an exam in advance and study those, then your final score on the exam isn't really representative.