News for hackers. Stuff that matters.
Re:Understanding, using, and finetuning Gemma by behnamoh on Saturday February 24, 2024 @09:47AM (#39491819)
Gemma (and Gemini) are heavily nerfed. Why are they on the news lately?

Also, Gemma is a +9B model. I think it's not okay that Google compared it with Mistral and Llama 2 (7B) models.

Google also took llama.cpp and used it in one of their Github repos without giving credit. Again, not cool.

All this hype seems to be backed by Google to boost their models whereas in practice, the models are not that good.

Google also made a big claim about Gemini 1.5 1M context window, but at the end of their article they said they'll limit it to 128K. So all that 1M flex was for nothing?

Not to mention their absurd approach in alignment in image creation.

Re:Understanding, using, and finetuning Gemma by htrp on Saturday February 24, 2024 @11:38AM (#39492858)
> Google also took llama.cpp and used it in one of their Github repos without giving credit. Again, not cool.

They said it was inspired by llama

>This is inspired by vertically-integrated model implementations such as ggml, llama.c, and llama.rs.

from https://github.com/google/gemma.cpp

Re:Understanding, using, and finetuning Gemma by pests on Sunday February 25, 2024 @02:26AM (#39498448)
Re:Understanding, using, and finetuning Gemma by d-z-m on Saturday February 24, 2024 @12:07PM (#39493139)
> Also, Gemma is a +9B model. I think it's not okay that Google compared it with Mistral and Llama 2 (7B) models.

They say it's because they're not counting embedding parameters[0]. Although apparently even with the embedding parameters subtracted it still rounds to 8B not 7B. From what understand, rounding to the nearest B is the standard. Seems slightly disingenuous to call it 7B, but not a big deal IMO since I don't hear anyone saying this model is outperforming popular OSS 7Bs.

[0]: https://huggingface.co/google/gemma-7b/discussions/34

Re:Understanding, using, and finetuning Gemma by sivakon on Saturday February 24, 2024 @10:43AM (#39492352)
It’s objectively worse in my local tests compared to Mistral. Again their model doesn’t include MT-bench benchmark because it’s really really bad at answering a follow up question(s). (this is also a problem in Ultra). It’s reasoning is also pretty bad compared to mistral.
Re:Understanding, using, and finetuning Gemma by b33j0r on Saturday February 24, 2024 @12:38PM (#39493417)
I can’t get it to recognize the stop token consistently in the 7b models.

About 50% of the shots, I get a sentence and a half of beautiful poetry, then a codeswitch into kanji, and then ral ral ral ral ral 膳 ral 杯 ral ral

Until I kill the process. Not every time, but way more often than the other llamas (which is basically never, these days).

I think they underestimated the impact of training on bulleted lists. It seems to love those!

Re:Understanding, using, and finetuning Gemma by rasbt on Saturday February 24, 2024 @10:04AM (#39491957)
> Gemma is a +9B model

Yes that's correct. It's 9.3B parameters if you count the embedding layer and final projection layer separately. However, since they used weight tying, the adjusted count is 8.5B as discussed in the article.

Re:Understanding, using, and finetuning Gemma by neodymiumphish on Saturday February 24, 2024 @10:38AM (#39492280)
Which still rounds to 9B and is 21.4% larger.
Re:Understanding, using, and finetuning Gemma by rasbt on Saturday February 24, 2024 @10:42AM (#39492330)
Yes, it's definitely unfair to count it as a 7B model. In that case, we could call Llama 2, which is 6.6B parameters, a 6B (or even 5B) parameter model.
Re:Understanding, using, and finetuning Gemma by neodymiumphish on Saturday March 02, 2024 @11:52AM (#39573790)
Except 6.6 rounds to 7. That’s completely reasonable. Arguing otherwise is pedantic.
Re:Understanding, using, and finetuning Gemma by gliptic on Saturday February 24, 2024 @09:59AM (#39491911)
> Google also took llama.cpp and used it in one of their Github repos without giving credit. Again, not cool.

Are you talking about gemma.cpp? Then no, they didn't.

Re:Understanding, using, and finetuning Gemma by andy99 on Saturday February 24, 2024 @10:01AM (#39491922)
Presumably he means this https://cloud.google.com/blog/products/application-developme...

The claim is correct but not related to gemma

Re:Understanding, using, and finetuning Gemma by gliptic on Saturday February 24, 2024 @10:09AM (#39492014)
From the repo:

> Run LLMs locally on Cloud Workstations. Uses:

> Quantized models from [Huggingface]

> llama-cpp-python's webserver

But sure, the blog post doesn't mention it.

Re:Understanding, using, and finetuning Gemma by behnamoh on Saturday February 24, 2024 @10:01AM (#39491927)
Yes, I was talking about this.
Re:Understanding, using, and finetuning Gemma by brucethemoose2 on Saturday February 24, 2024 @10:26AM (#39492152)
Counterpoints:

- Local models are pretty easy to de-censor, if thats what you mean.

- ...Yeah, it should not be labeled as a 7B. Its sort of 7B class.

- The repo mentions they use the llama-cpp-python server

- 1M context brute forced across TPUs is insanely expensive, I can see why Google reigned it in.

But overall your message is not wrong. Google is hyping Gemma a ton when its... Well, not very remarkable. And they could have certainly made something niche and interesting, like a long context 8.5B model, a specialized model, a vastly more multilingual model, something to differentiate it from Mistral 7B 0.2

Re:Understanding, using, and finetuning Gemma by andy99 on Saturday February 24, 2024 @09:54AM (#39491866)
Gemma has a 7B parameters model https://huggingface.co/google/gemma-7b that's what I saw compared to Mistral

(Edit: I'm wrong)

Re:Understanding, using, and finetuning Gemma by light_hue_1 on Saturday February 24, 2024 @09:59AM (#39491912)
No it doesn't.

Gemma 7B is a 9B model. The name is a lie. Then they really played games with Gemma 2B as well.

I don't get how Google can be this incompetent and far behind everyone else. They have amazing people and the kinds of resources that almost no one else does but somehow need to resort to faking demos, blatant lies about model sizes, etc.

Google used to be the place everyone wanted to go. Someone at Google AI needs to be fired so they can start being productive again.

Re:Understanding, using, and finetuning Gemma by xfalcox on Saturday February 24, 2024 @10:22AM (#39492127)
> Gemma 7B is a 9B model. The name is a lie

Ohhh so that explains why I couldn't load it on my RTX 4090, while other 7B models load just fine!

Re:Understanding, using, and finetuning Gemma by andy99 on Saturday February 24, 2024 @10:02AM (#39491944)
Haha really? I almost added the caveat that I didn't count the parameters myself. And I couldn't see the weights file size because it requires login (because of their restrictive licensing choice). If true and it's 9B that's really dishonest.
Re:Understanding, using, and finetuning Gemma by rasbt on Saturday February 24, 2024 @10:14AM (#39492057)
Yes, it's 8.5B params if you account for weight tying, and 9.3B if you count the embedding layer and output layer weights separately as shown in the 2nd figure in the article. In the paper, I think they justified 7B by only counting the non-embedding parameters (7,751,248,896), which is kind of cheating in my opinion, because if you do that, then Llama 2 is basically a 5B-6B param model.
Re:Understanding, using, and finetuning Gemma by andy99 on Saturday February 24, 2024 @11:23AM (#39492717)
Is the 2B measured like that as well? I did use it with llama.cpp and noticed it ran slower than I expected.

That's the danger of too much abstraction, it's easy to have big gaps in one's understanding of what's really going on.

Re:Understanding, using, and finetuning Gemma by rasbt on Saturday February 24, 2024 @11:36AM (#39492841)
Yes, it's somewhat similar to the 2B model as it uses the same vocabulary size.
Re:Understanding, using, and finetuning Gemma by brucethemoose2 on Saturday February 24, 2024 @10:21AM (#39492117)
Practically, speaking, I OOM'd running Gemma on a 3090 using a config that had VRAM to spare for Mistral 7B. It kinda surprised me at first, until I realized why.
Re:Understanding, using, and finetuning Gemma by cyanydeez on Saturday February 24, 2024 @10:22AM (#39492128)
the context window is entirely limited by VRAM size

do you even LLM?