> So with that in mind, how can we reject the possibility that as an AI Agent (e.g. Aider) works on a program over time, it bootstraps a theory?

Lack of effective memory, that might have worked if you constantly retrained the LLM incorporating the new wisdom iteratively like a human does, but current LLM architecture doesn't enable that. The context provided is neither large enough nor can it use it effectively enough for complex problems.

And this isn't easy to solve, you very quickly collapse the LLM if you try to do this in the naive ways. We need some special insight that lets us update LLM continuously as it works in a positive direction the way humans can.

Yeah, that's a good point. I absolutely agree that it needs access to effective long-term memory, but it's unclear to me that we need some "special insight". Research is relatively early on this, but we already see significant sparks of theory-building using basic memory retention, when Claude and Gemini are asked to play Pokemon [0][1]. It's clearly not at the level of a human player yet, but it (particularly Gemini) is doing significantly better than I expected at this stage.

[0] https://www.twitch.tv/claudeplayspokemon

[1] https://www.twitch.tv/gemini_plays_pokemon

They update that gemini plays pokemon model when it gets stuck with new prompt engineering etc. So there the learning happens by a human and not the LLM, the LLM can do a lot with trial and error but if you follow it there it does the same action over and over and get stuck until the prompt engineering kicks it into self evaluating 20 steps later.

So that isn't just "ask it to play pokemon", that is a large program with tons of different prompts and memories that kicks in at different times, and even with all that and updates to the program when it gets stuck it still struggles massively and repeats mistakes over and over in ways human never would.

Yes, indeed. They think that every circular argument is vicious. Not at all, there are two kinds of circularity: virtuous circularity; vicious circularity. Bootstrapping falls under the former. Check [1] and [2]

[1] https://www.hipkapi.com/2011/03/10/foundationalism-and-virtu...

[2] Brown, Harold I. “Circular Justifications.” PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association 1994 (1994): 406–14. http://www.jstor.org/stable/193045.

The information needs to propagate through the network either forward (when the model has the codebase in context) or backward (when it updates its weights).

You can have the models pseudo “learn” by putting things in something like a system prompt but this is limited by context, and they will never permanently learn. But we don’t train at inference time with today’s LLMs.

We can explicitly reject this possibility by looking at the information that goes into the model at train and test time.

> So with that in mind, how can we reject the possibility that as an AI Agent (e.g. Aider) works on a program over time, it bootstraps a theory?

That's the appropriate level of faith for today's LLMs. They're not good enough to replace programmers. They're good enough that we can't reject the possibility of them one day being good enough to replace programmers.

And good enough does not mean "as good as". Companies happily outsource programming jobs to worse, but much cheaper, programmers, all the time.

I for one wouldn't mind seeing more focus on probability than possibility here.

Possibility means practically nothing.