News for hackers. Stuff that matters.

A comment on The Benchmarkpocalypse.

Re:The Benchmarkpocalypse by stephantul on Tuesday August 18, 2026 @01:59AM (#49341838)
Unfortunately, even a holdout set doesn’t protect you from overfitting, it just takes longer.

Of course having a holdout set is better than not having one. It’s just not a silver bullet.

[ Parent | on HN ]
Re:The Benchmarkpocalypse by frumiousirc on Tuesday August 18, 2026 @06:36AM (#49343707)
That is not what I read from danluu's words. He merely stated in the prompt that there is a holdout set and did not iterate to minimize error against the holdout set. In a prior attempt he prompted with only "don't overfit" to ill effect on the holdout eval. Did I misread?
[ Parent | on HN ]