News for hackers. Stuff that matters.
Re:StarCoder and StarCoderBase: 15.5B parameter models with ... by mrtranscendence on Tuesday May 16, 2023 @11:17AM (#35963002)
That paper does show evidence of diminishing returns, for what it’s worth. You get less going from 64 to 540 than you do from 8 to 64. Combined with the increased costs of training gargantuan models, it’s not clear to me that models with trillions of parameters will really be worth it.

There are no comments on this comment yet.