News for hackers. Stuff that matters.
Re:ARC-AGI without pretraining by Ukv on Tuesday March 04, 2025 @04:44PM (#43260106)
> there is a lot of evidence that many of the concepts ARC-AGI is (allegedly) measuring are innate in humans

I'd argue that "innate" here still includes a brain structure/nervous system that evolved on 3.5 billion years worth of data. Extensive pre-training of one kind or another currently seems the best way to achieve generality.

Re:ARC-AGI without pretraining by onemoresoop on Tuesday March 04, 2025 @09:13PM (#43261799)
Sure and LLMs ain’t nothing of this sort. While they’re an incredible feat in technology, they’re just a building block for intelligence, an important building block I’d say.
Re:ARC-AGI without pretraining by SiempreViernes on Tuesday March 04, 2025 @09:24PM (#43261867)
So all billons spent finding tricks and architectures that perfom well haven't resulted in any durable structures in contemporary LLMs?

Each new training from scratch is a perfect blank slate and the only thing ensuring words come out is the size of the corpus?

Re:ARC-AGI without pretraining by Ukv on Wednesday March 05, 2025 @05:22AM (#43264938)
> Each new training from scratch is a perfect blank slate [...]?

I don't think training runs are done entirely from scratch.

Most training runs in practice will start from some pretrained weights or distill an existing model - taking some model pretrained on ImageNet or Common Crawl and fine-tuning it to a specific task.

But even when the weights are randomly initialized, the hyperparameters and architectural choices (skip connections, attention, ...) will have been copied from previous models/papers by what performed well empirically, sometimes also based on trying to transfer our own intuition (like stacking convolutional layers as a rough approximation of our visual system), and possibly refined/mutated through some grid search/neural architecture search on data.