News for hackers. Stuff that matters.
Re:StarCoder and StarCoderBase: 15.5B parameter models with ... by coldtea on Tuesday May 16, 2023 @07:12AM (#35960163)
>Google trained 3 differently sized models of the same architecture (8b, 62b, 540b) on the same dataset of 780b tokens and evaluated all 3 on various tasks.

That's quite a small sample to argue the generic point that "for any arbitrary performance x, the data required to reach it reduces with size".

Key part being: "for any arbitrary performance".

Re:StarCoder and StarCoderBase: 15.5B parameter models with ... by og_kalu on Tuesday May 16, 2023 @07:41AM (#35960414)
Any paper that's trained more than one model size on the same data affirms the same thing.

Llama 13b was better than 7b and Llama 66b was better than 33b.

Literally pick any one.