Right. Gemini Pro 1.5 scores 81.9% on MMLU and is also above in a few other benchmarks.

Which - importantly - is better than Mistral at 81.2% ...

Gemini Ultra scored 90% which is better than GPT-4.

This reads like a paid-for press release from Microsoft to pretend like they're almighty and Google is incompetent.

Didn't Gemini 1.0 Ultra turn out to be worse than GPT 4 Turbo in practice? Synthetic benchmarks aren't a perfect representation of anything.

I think they're close enough that it depends on your use case. IME, which is mostly generating Python and shell code, Gemini Advanced is faster, less lazy and generates overall better solutions than GPT-4. It also seems to be more up-to-date with libraries and stuff. It can also directly go to URLs (e.g. summarise a paper), which GPT-4 refuses to do.

Doesn't MMLU have like 2-3% wrong answers anyways ?

With 128K context (1M paid) compared to 32K. Man, Google is going to be a game changer for especially free AI tiers.

Edit: BTW, more Mistral benchmarks here: https://docs.mistral.ai/platform/endpoints/ TIL Mistral Small outperforms Mixtral 8x7B.

1.5 was probably released too late to be tested.

Is 1.5 even publicly available yet?

Yes

How do I use it? I only have access to 1.0 Ultra through Gemini Advanced.

Me too, I don't think it's public yet.

And that doesn't count. Only pro 1.0 available for me through APIs. I need to be able to test for myself the capabilities.

As it stands best LLM available by API by Google is far behind GPT4.

it’s not public, there is waitlist

But Ultra 1.0 is available to compare against, right?

How can I get access to it? The best I can compare against is pro 1.0.

Most of my usecases are logic based on embedded content in the prompt and nothing available to me beats GPT-4 there.

Ultra 1.0 is theoretically available via API, but it's only available to allowlisted customers, which seems to be a very small number right now. It's not generally available.

Ultra 1.0 is the model behind Gemini Advanced, which is generally available.

Not via an API, which is needed to perform proper tests.

From the article:

> generally available through an API (next to GPT-4)