Like law of conservation energy, matter and momentum, I guess there must be law of conservation of knowledge in context of LLMs?

Oof I really don’t think there must be such a law. The basic physics of the universe and the mathematical structure of a human engineered LLM are just very different things.

There must be. Since there is a finite amount of real world reward signal captured by these models, there can only be a finite amount of grounded knowledge that can come out. Usually without reasoning, the output knowledge "yield" will be far lower than input.

The interesting piece is if we can get LLMs to reason well, that finite reward capture can take LLMs much farther than possibly humans could do with the same signal.

The "compressed" knowledge is fixed. But reasoning can create new knowledge, that's what humans do all the time.

You can't do better than perfect. But if the default is sufficiently bad, you can seem to violate conservation by putting more effort in.

Similarly to when a machine has an efficiency of 1%, optimizing it to 10% does not violate thermodynamics.

There is a limit to what you can learn from limited knowledge. Every bit of information you learn can only divide the space of possible theories by half. However, current LLMs are ludicrously far from that limit.

My Pareto law senses are tingling.

Nope! That surely doesn't apply to humans, why would it necessarily apply to learning machines?

The human equivalent would be the process of armchair philosophizing. Humans get around the limit by being able to go and look at the world to obtain more knowledge -- if you didn't let humans look at the world, they too would have a "knowledge limit".

And if you _do_ let LLMs look at the world, and train on that, they also wouldn't have that limitation.

> one can produce high quality training data with the LLM itself which the next iteration of that model can use. Rinse and repeat.

it may be higher quality, but far from perfection.

After many rinses and repeats dataset will accumulate significant amount of errors.

That depends. You can implement an error checking algorithm with the LLM itself to check for drift in meaning and loss of original context from version to version as you improve your training dataset, as well as measure performance as you train new versions of your LLM on that dataset.