Gemma 2: Improving Open Language Models at a Practical Size [pdf]

Re:Gemma 2: Improving Open Language Models at a Practical Si... by luke-stanley on Thursday June 27, 2024 @12:15PM (#40812141)

It's fairly easy to pay OpenAI or Mistral money to use their API's. Figuring out how Google Cloud Vertex works and how it's billed is more complicated. Azure and AWS are similar in how complex they are to use for this. Could Google Cloud please provide an OpenAI compatible API and service? I know it's a different department. But it'd make using your models way easier. It often feels like Google Cloud has no UX or end-user testing done on it at all (not true for aistudio.google.com - that is better than before, for sure!).

[ Parent | on HN ]

Re:Gemma 2: Improving Open Language Models at a Practical Si... by alekandreev on Thursday June 27, 2024 @12:19PM (#40812188)

Happy to pass on any feedback to our Google Cloud friends. :)

[ Parent | on HN ]

Re:Gemma 2: Improving Open Language Models at a Practical Si... by luke-stanley on Thursday June 27, 2024 @12:47PM (#40812464)

Thank you!

[ Parent | on HN ]

Re:Gemma 2: Improving Open Language Models at a Practical Si... by anxman on Thursday June 27, 2024 @05:42PM (#40815445)

I also hate the billing. It feels like configuring AWS more than calling APIs.

[ Parent | on HN ]

Re:Gemma 2: Improving Open Language Models at a Practical Si... by hnuser123456 on Thursday June 27, 2024 @12:33PM (#40812336)

I plan on downloading a Q5 or Q6 version of the 27b for my 3090 once someone puts quants on HF, loading it in LM studio and starting the API server to call it from my scripts based on openai api. Hopefully it's better at code gen than llama 3 8b.

[ Parent | on HN ]

Re:Gemma 2: Improving Open Language Models at a Practical Si... by bapcon on Thursday June 27, 2024 @12:35PM (#40812350)

I have to agree with all of this. I tried switching to Gemini, but the lack of clear billing/quotas, horrible documentation, and even poor implementation of status codes on failed requests have led me to stick with OpenAI.

I don't know who writes Google's documentation or does the copyediting for their console, but it is hard to adapt. I have spent hours troubleshooting, only to find out it's because the documentation is referring to the same thing by two different names. It's 2024 also, I shouldn't be seeing print statements without parentheses.

[ Parent | on HN ]

Re:Gemma 2: Improving Open Language Models at a Practical Si... by logankilpatrick on Friday June 28, 2024 @08:14AM (#40819866)

We are working hard to improve this across ai.google.dev (Gemini API), Hang tight!

[ Parent | on HN ]

Re:Gemma 2: Improving Open Language Models at a Practical Si... by ankeshanand on Thursday June 27, 2024 @01:36PM (#40812969)

If you're an individual developer and not an enterprise, just go straight to Google AIStudio or GeminiAPI instead: https://aistudio.google.com/app/apikey. It's dead simple getting an API key and calling with a rest client.

[ Parent | on HN ]

Re:Gemma 2: Improving Open Language Models at a Practical Si... by luke-stanley on Thursday June 27, 2024 @03:55PM (#40814393)

Interesting but when I tried it, I couldn't figure out the billing model because it's all connected to Google projects, and there can be different billing things for each of them.

Each thing seems to have a bunch of clicks to setup that startup LLM providers don't hassle people with. They're more likely to just let you sign in with some generic third party oAuth, slap on Stripe billing, let you generate keys, show you some usage stats, getting started docs, with example queries and a prompt playground etc.

What about the Vertex models though? Are they all actually available via Google AI Studio?

[ Parent | on HN ]

Re:Gemma 2: Improving Open Language Models at a Practical Si... by lhl on Thursday June 27, 2024 @05:31PM (#40815351)

Sadly, while gemma-2-27b-it is available (as a Preview model) on the AI Studio playground, it didn't show up via API on list_models() for me.

[ Parent | on HN ]

Re:Gemma 2: Improving Open Language Models at a Practical Si... by Deathmax on Thursday June 27, 2024 @01:38PM (#40812996)

Gemini models on Vertex AI can be called via a preview OpenAI-compatible endpoint [1], but shoving it into existing tooling where you don't have programmatic control over the API key and is long lived is non-trivial because GCP uses short lived access tokens (and long-lived ones are not great security-wise).

Billing for the Gemini models (on Vertex AI, the Generative Language AI variant still charges by tokens) I would argue is simpler than every other provider, simply because you're charged by characters/image/video-second/audio-second and don't need to run a tokenizer (if it's even available cough Claude 3 and Gemini) and having to figure out what the chat template is to calculate the token cost per message [2] or figure out how to calculate tokens for an image [3] to get cost estimates before actually submitting the request and getting usage info back.

[1]: https://cloud.google.com/vertex-ai/generative-ai/docs/multim...

[2]: https://platform.openai.com/docs/guides/text-generation/mana...

[3]: https://platform.openai.com/docs/guides/vision/calculating-c...

[ Parent | on HN ]

Re:Gemma 2: Improving Open Language Models at a Practical Si... by luke-stanley on Thursday June 27, 2024 @03:59PM (#40814443)

Good to know about this API preview. Hopefully the billing problem and UI maze of Vertex AI can be sorted too?

[ Parent | on HN ]

Re:Gemma 2: Improving Open Language Models at a Practical Si... by Flumio on Thursday June 27, 2024 @05:15PM (#40815209)

Google does plenty of ux studies on gcp. I took part in at least 3 of them.

I'm also not sure if I understand your problem with pricing? Depending on what you do with it, it's not just an LLM. It actually started before llms.

Pricing for image classification and other features are completely different products like an LLM.

[ Parent | on HN ]

Re:Gemma 2: Improving Open Language Models at a Practical Si... by luke-stanley on Thursday June 27, 2024 @06:01PM (#40815641)

They should do a whole lot more then! Ideally they'd have effective impact. It's a busy mess on GCP. If they wanted to compete well, they should do much better with UX design, especially for onboarding. Compare how easy setting up a Mistral account is with GCP to do some generative LLM in a Python script. GCP is a maze. Did you make an account to reply to this? I'm curious what you do with GCP? Are you a heavy user?

[ Parent | on HN ]

Re:Gemma 2: Improving Open Language Models at a Practical Si... by Flumio on Friday June 28, 2024 @02:54AM (#40818354)

I create new accounts because I use hn too much.

I use gcp professional every day and always found it quite intuitive.

Did plenty of image classification with vertex ai too

[ Parent | on HN ]

Re:Gemma 2: Improving Open Language Models at a Practical Si... by luke-stanley on Friday June 28, 2024 @07:51AM (#40819702)

Why would you make new accounts because you use HN too much? Doesn't make sense to me. Anyhow if you use GCP every day, you're going to have learned it's weird clunky behaviour. GCP's main problem is that they've steadily become a sprawling mess of complexity, which is in big contrast to quite a few LLM specific cloud services that are happy to take peoples money without extra complexity?

[ Parent | on HN ]

Re:Gemma 2: Improving Open Language Models at a Practical Si... by Flumio on Friday June 28, 2024 @08:21AM (#40819911)

Not being logged in feels like a bigger hurdle to comment and check if someone responded to it.

It's a shitty solution to a stupid problem ;)

But I did mention that vertex AI is more than just hosting llms though

[ Parent | on HN ]