News for hackers. Stuff that matters.
Re:Understanding, using, and finetuning Gemma by rasbt on Saturday February 24, 2024 @10:04AM (#39491957)
> Gemma is a +9B model

Yes that's correct. It's 9.3B parameters if you count the embedding layer and final projection layer separately. However, since they used weight tying, the adjusted count is 8.5B as discussed in the article.

Re:Understanding, using, and finetuning Gemma by neodymiumphish on Saturday February 24, 2024 @10:38AM (#39492280)
Which still rounds to 9B and is 21.4% larger.
Re:Understanding, using, and finetuning Gemma by rasbt on Saturday February 24, 2024 @10:42AM (#39492330)
Yes, it's definitely unfair to count it as a 7B model. In that case, we could call Llama 2, which is 6.6B parameters, a 6B (or even 5B) parameter model.
Re:Understanding, using, and finetuning Gemma by neodymiumphish on Saturday March 02, 2024 @11:52AM (#39573790)
Except 6.6 rounds to 7. That’s completely reasonable. Arguing otherwise is pedantic.