evilmathkid

well not if its an open textbook exam

in non metaphor terms: In many ML situations you can carry the train set with you test time. Eg: KNN, SVM, replay buffers, etc

this is one such case

--

the separate overfitting concern is fair, look at private holdout performance for that. it performs on par with TRM (a comparable model) in the private set, ofc with far less compute

Yeah agree with pg completely. Multiple people I know did this

Yeah but I posted this work back in Dec-Feb lol

no, most models on kaggle are finetuning during test time, (including LLM based approaches)

Pure frontier LLMs dont, but thats because nobody knows how to make it work cleanly and at scale. Once someone makes it work, it will be deployed

Not true. This is allowed in a metalearning context. Its called transductive learning and has existed since the 90s: https://en.wikipedia.org/wiki/Transduction_(machine_learning...

I address this in more detail in the blog

I didn't phrase that well and cant edit, so clarifying:

What I did was "You are born during the exam, given access to a training set (which is curated and allowed) and the questions then learn everything from scratch during the exam"

what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam"

I put more details in the answer here: https://news.ycombinator.com/item?id=49525841

2 different things are being called leakage here

1) weight update during eval: this is a form of test time training and not really cheating. It is also closer to Sutton's views of intelligence: models should learn during deployment, instead of being frozen after training

2) overfitting: agree that true measure is private set. It scores on private set roughly on par with TRM (a comparable model), obv with much lesser compute

--

also re LLMs: they do not follow this 3 split since (a) Incredibly hard to keep a pretrain dataset clean, (b) common in labs to benchmaxx during postraining (and known to do so on ARC)

haha thanks!

As of today yes I agree with you

in the future, for general perf, I am optimistic that someone will figure out an alphazero like approach (ilya/silver/sutton/carmack seem to be working on something like this)

train from scratch only during the 12 hours allowed on Kaggle

Other competitions have implemented things like this before. Eg: OpenAI's Parameter Golf and Keller Jordan's Modded NanoGPT Speedrun

Glad to know you like it!

1) Unfortunately I didn't. I was v new to ML when I did this and didnt have time or skill to try many things. Will try them when I get some time!

2) Possibly, but it would require significant changes and effort. But much larger models would be required imo (must have capacity greater than the complexity of the problem)

> You mean for a specific usecase?

Yepp

> Also aren't frontier models trained 'from scratch'?

The full sentence was supposed to be "training from scratch only on ARC data"

the point was you dont need large scale pretraining

> In the university I first dropped out of, students that surpassed me studied by getting and sharing copies of previous exams and solving those question

Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam"

What you described is cheating because more time than the exam permits. ARC was designed specifically to avoid this. The exam in question (kaggle competition) is 12hrs long with 4xL4s. I trained on 1.5hrs with a single 5090 (which converted to 4xL4s is slightly longer, but still within 12hrs).

(There's also access to experts who know the answer, which kaggle bans by banning the internet)

Lucas describes it well here (and his original tweet up the thread): https://x.com/giffmana/status/2002128356901597509

--

Your arguments btw support my work over the LLMs more. LLMs today are postrained with a large amount of synthetic ARC data. (Exactly the "teach to test" criticism). Thats why they perform so well on ARC. Base models still are terrible at ARC-2

same! I'd often learn during the exam by solving an easier problem and then that would let me tackle a hard problem

Thanks!

I think this is a great question. I have some thoughts on this but no hard evidence (neither does anyone else!)

Your argument relies on the AGI system being the model arch + weights. I think that the weights are irrelevant. The training algorithm is what is AGI: You choose/find a training set that covers a task, and then some form of deep learning with a big neural net.

For LLMs (which is a weak kind of AGI), this is freezing a model after NTP pretraining + RL postraining. This allows us to do well on a wide distribution of tasks.

But if we had a scoped task, I think its possible to just do the same thing with (1) a smaller dataset that covers this task + (2) not freezing the model (ie. test time training). This is easy for ARC because the data is small and has been curated well yes, but i don't see why can't this apply to more complex/ill-defined tasks (obv lots of things unsolved to make it work today)

> even if the ARC train+eval sets contain the sum of human intuition between them, superficial differences IRL would render your model unable to identify which examples are relevant to which problems, and thus unable to transductively reason.

In this hypothetical world, the dataset becomes incredibly large, and training on it makes it close to an LLM. You can then finetune transductively and we get the same thing (other approaches already do this iirc)

--

Note: BTW I'm not claiming that this is an AGI system, the benchmark creator also was clear that ARC is not sufficient for AGI (his goal was just to point out unsolved stuff, and the stuff ARC-2 pointed out was demonstrated very clearly by large reasoning models).

In the above work, I just wanted to show that AR transformers without pretraining can perform really well on this benchmark, which was incredibly non-obvious before.

The claims in this argument are separate and I haven't proved them yet

there's a very large variance in doctors' abilities in India. At the very top they are close to the best in the world, esp with an insanely high workload.

but on an average, not great

Also, a lot of gymgoers and physical trainers I know hadn't heard of rhabdo either (and this is a relatively wealthy part of a tier 1 city)

things are changing for the better however

9 color tokens + 4 special tokens (start, end, newline, inp_out_sep)

Nope, it never learns how it did on the questions.

During test time, you have to submit all the answers at once and you get the total score (so you dont even know which puzzles were solved)

thanks! was incredibly scary when it happened

I'm currently 10th in the world on the private set on Kaggle. And iirc, at one point I was 4th

Can't comment more since its an ongoing competition

What you gather is correct, assuming by "the test" you mean the ARC benchmark in general. It was controversial because people are used to LLMs which are frozen at train time, where the eval problems are usually not trained on for various reasons like fragility (basically porridgeraisin's ans which is great)

Here's another explanation. Take the train dataset and test dataset of a benchmark

Train: {x_i -> f(x_i)}, Test: {x_j -> f(x_j)}

As long as f(x_j) in the test set is hidden, there is no "training on test". In a normal benchmark, each x_i is a single datapoint. But in metalearning benchmarks like ARC, x_i is the puzzle itself that has a train set and the test questions within it, hence the confusion and controversy

idk the definition is fuzzy. thats why people use the "modern" qualifier to talk about decoder-only style and this is also not clean since you now have reasoning models which are separate

its not gonna do well on ARC-3 without some significant changes and effort

The new arch in that video is kinda misleading. Didn't really compare against proper baselines

> The "67 cents" part though is misleading imho. You can't extrapolate from there and think that investing say $100 will get you a lot better results.

I think thats unfair. Perf-compute is often logarithmic and will always saturate . Reaching the plateau faster is valuable as it often leads to better peaks (held true here and also look at modded nanogpt)

And more compute increases the perf (after dealing with other scaling problems)

Yeah I've reached much higher perf but

- it feels logarithmic (like most perf-compute graphs), and eventually plateaus. 44% @ 67 cents was a good stopping point for me

- more compute would require a lot of effort and dealing with new problems like training stability, cost of iterations/sweeps (didnt have the money to convincingly run larger iterations)

Its not an LLM if there's no pretraining. AR transformers were around before LLMs and will be there after LLMs.

When I made this, the point was to show that you dont need pretraining (which is what makes an LLM) to perform well on complex tasks

And yes it is not a language model either. I did not train it on any language data. Only ARC puzzles

Hi! Author here. Surprised to see this on HN now. Happy to answer any questions!

Some context about this:

- This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs

- Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but used v complex architectures or extremely high amounts of training compute. No one expected a simple AR transformer to perform this well, at this low cost and w these few training samples.

- Sample Efficiency is one of the most important unsolved problems today in AI. That's what I was targetting with this work. We know it is easy to increase SE by increasing compute/params, so it was important to constrain cost as much as possible (also why OpenAI's Parameter Golf had fixed compute and why Modded NanoGPT is considered very sample efficient)

- Can the perf be improved? Yes but the competition is ongoing so can't talk about it

- Personally I think today's frontier models can be beat by training from scratch. Haven't proved this yet tho

- Fun: I was new to ML when I posted this first (dec '25). I basically used ARC as a way to learn ML

This is a small transformer trained from scratch in 1.5hrs on a 5090 that beats many LLMs. Code is open source.

I want to solve sample efficiency and this work is an attempt to find the limits of transformers and today's methods while keeping costs low so I can iterate fast.

Blog: https://mvakde.github.io/blog/44-on-arc-1/ X thread: https://x.com/evilmathkid/status/2029519274835148829

(This is the public eval. Also it scores 7% on ARC-2)