News for hackers. Stuff that matters.
Re:ARC-AGI without pretraining by pona-a on Tuesday March 04, 2025 @04:39PM (#43260067)
Would that make in-context learning a superset or a subset of pretraining?

This paper claimed transformers learn a gradient-descent mesa-optimizer as part of in-context learning, while being guided by the pretraining objective, and as the parent mentioned, any general reasoner can bootstrap a world model from first principles.

[0] https://arxiv.org/pdf/2212.07677

Re:ARC-AGI without pretraining by ta8645 on Tuesday March 04, 2025 @05:00PM (#43260260)
> Would that make in-context learning a superset or a subset of pretraining?

I guess a superset. But it doesn't really matter either way. Ultimately, there's no useful distinction between pretraining and in-context learning. They're just an artifact of the current technology.