News for hackers. Stuff that matters.
Re:ARC-AGI without pretraining by pona-a on Tuesday March 04, 2025 @05:10PM (#43260336)
In-context learning is a very different problem from regular prediction. It is quite simple to fit a stationary solution to noisy data, that's just a matter of tuning some parameters with fairly even gradients. In-context learning implies you're essentially learning a mesa-optimizer for the class of problems you're facing, which in the form of transformers means essentially means fitting something not that far from a differentiable Turing machine with no inductive biases.

There are no comments on this comment yet.