Re:Understanding, using, and finetuning Gemma
by neodymiumphish on Saturday February 24, 2024 @10:38AM
(#39492280)
Which still rounds to 9B and is 21.4% larger.
Re:Understanding, using, and finetuning Gemma
by rasbt on Saturday February 24, 2024 @10:42AM
(#39492330)
Yes, it's definitely unfair to count it as a 7B model. In that case, we could call Llama 2, which is 6.6B parameters, a 6B (or even 5B) parameter model.
Re:Understanding, using, and finetuning Gemma
by neodymiumphish on Saturday March 02, 2024 @11:52AM
(#39573790)
Except 6.6 rounds to 7. That’s completely reasonable. Arguing otherwise is pedantic.
Yes that's correct. It's 9.3B parameters if you count the embedding layer and final projection layer separately. However, since they used weight tying, the adjusted count is 8.5B as discussed in the article.