News for hackers. Stuff that matters.

A comment on Andrew Ng: Unbiggen AI. Parent comment

Re:Andrew Ng: Unbiggen AI by riku_iki on Thursday April 07, 2022 @12:19AM (#30940560)
> it is possible to smartly choose the correct few samples for the network that yield the same results.

And then, someone is using pretrained 500B model, and fine-tuning your few examples, and getting new SOTA.

[ Parent | on HN ]
Re:Andrew Ng: Unbiggen AI by YeGoblynQueenne on Thursday April 07, 2022 @08:25AM (#30943524)
They might get new SOTA because the metric is accuracy, but if the metric was accuracy weighted by sample efficiency, then SOTA would look a lot less impressive.

Simplest way to weigh by sample efficiency: multiply accuracy by ratio of test set to training set sizes. Everyone's training/testing on 80/20 splits, so everybody's SOTA would go down by 3/4s.

[ Parent | on HN ]