It won't be available to regular devs until Q2 next year probably (January for selected partners). So they are roughly a year behind OpenAI - and that is assuming their model is not overtrained to just pass the tests slightly better than GPT4

> and that is assuming their model is not overtrained to just pass the tests slightly better than GPT4

You are assuming GPT4 didn't do the exact same!

Seriously, it's been like this for a while, with LLMs any benchmark other than human feedback is useless. I guess we'll see how Gemini performs when it's released next year and we get independent groups comparing them.

>So they are roughly a year behind OpenAI

Possibly by that time GPT5 will already be out.

Possibly, but I doubt it! I'd expect a response by OpenAI via GPT-4V improvements soon though.

I would not underestimate DeepMind with its access to Google's resources and private data which OpenAI lacks, even with Microsoft's help.

They already caught up and surpassed GPT-4 and OpenAI's availability and APIs are very unstable and all that matters is that and the cost per token.

>Finally, some competition for GPT4 API!!! This is such good news.

Save your enthusiasm for after it launches; Google's got a habit of over-promising when it comes to AI.

Everything they published thus far in the generative AI space has been abysmal in quality compared to the competition. I'd be hella surprised if this reaches GPT-4 levels of quality...

I'm a GPT4 subscriber and a Google GSuite work subscriber. I've been using the latest Bard this morning to write and refine python code, and it's just as good if not slightly better than GPT4. I asked it to refine some obtuse code with lots of chaining, and it did an admirable job writing accurate comments and explaining the chained logic. It's ridiculously anecdotal of course, but I used Bard for all of 5 minutes last time they announced. This time seems different.

Manifold has this at 69%, so here's an opportunity to take some people's internet points: https://manifold.markets/brubsby/will-googles-gemini-beat-gp...

Looks like it's 75%? It briefly dropped to 68% in October/November, but has been fairly consistently around 75% for a while.

It's very active today: 50+ trades in the last hour. When I checked it was 69%, but it's gone up and down since then. Click on the "trades" tab to see.

Ah fair enough, this seems to be an awkward UI. Over time though 75% seems to be the representative recent position. It is up to 85% now though!

I’m curious which instances of overpromising you’re referring to.

Like how much they hyped up Bard, which when released turned out to be barely competitive with GPT3.5. E.g. https://www.reuters.com/technology/google-ai-chatbot-bard-of...

I definitely think GPT is better than Bard, but Bard definitely did live up to the hype in a few ways. The two that blew my mind (and still do to some extent) are the blazing speed and the ability to pull information real time (no more pesky knowledge cutoff date). Bard also felt pretty comparable to 3.5 to me, better in some things and worse in others. Coding was definitely a bust with Bard.

ChatGPT via plugins and tools can access real time data, the bot I built for slack at my work has the ability to load Web pages, search DDG etc.

I do not recall Bard being said to be better than any particular other model, but then having worse performance by some metric when released.

Your link isn’t really an indication of an overpromise.

Bard isn't a model, it's a product. Saying comparisons against "Bard" without specifying a particular point in time are like analyses of "ChatGPT" without specifying a model. There have been a number of releases adding more features, tool use, making it smarter, and crucially adding more languages. ChatGPT is not fine-tuned in different languages – it manages them but lacks cultural context. That's one place Bard is quite far ahead from what I've seen.

all that shows is that google screwed up their positioning, and openai got it right

people don't see a difference between model and product, they think "gpt3 is ok", "gpt4 is great", "bard is like gpt3"

it's not the consumer's fault when the business has a positioning mistake, the business has to try and win the consumer back

Most people don't use LLMs. Of those that do most people just know they're using "ChatGPT". A slim minority care about the model.

In my opinion, not focusing on the model, focusing on the product, and focusing on positioning for normal users (free, fast, fine tuned in many languages, "easy"), is a better product positioning.

> In my opinion, not focusing on the model, focusing on the product, and focusing on positioning for normal users (free, fast, fine tuned in many languages, "easy"), is a better product positioning.

Does google agree? doesn't the fact that they're so deliberately creating user-focused branding for different models (ultra, pro, nano) show they also see the value in the differentiation?

I can't speak for Google, and must emphasise that these are personal opinions. However I'd say that this entire marketing push is mostly for the super-engaged early adopters, not targeted at the general public. Looking at the YouTube videos, the more they seem to be targeted towards a general audience the less they mention these specifics. So, I suspect that the Ultra/Pro/Nano branding will mostly be used on the advanced Bard product that they speak about in the launch blog post, and on the APIs available to developers.

In terms of AI? Last year? A better question is what wasn't an overpromise?

Heh, I read that and had the opposite conclusion.

When I was reading the benchmarks and seeing how Gemini Ultra was outperforming GPT-4 I thought, "Finally, some competition for GPT4"!

But when I got to that part, that's when I realized that it could potentially be caught in release hell and not actually see the light of day or significant use. Google, for better or worse, has more of a brand reputation to maintain and is more risk averse, so even if Gemini Ultra can, in theory, outperform GPT4, users might not get a chance to access it for a while.

"Google, for better or worse, has more of a brand reputation to maintain"

You think this is why Google is so far behind?

It absolutely is. Googlers here will know that there was an internal version of ChatGPT that got canned because of halucinations.

better than chatgpt? Or canned because hallucinations were even worse?

Isn’t that Bard?

Absolutely I do. Internally they have some incredible stuff, but the leadership is terrified of letting normies try it out because of the (real or perceived I don't know) damage to the brand that would happen if it said something racist or misogynist, etc.

No way, that's what they want you to think. The idea that Google would be behind technologically would be an embarrassment they can't handle. The 3.5 level gemini pro is probably just as capable of saying racist or misogynist stuff so there's no reason why they're allowing that to be public while the "GPT-4 beating" Ultra is hidden if it's just because of that. More likely Ultra is just not as good as these benchmarks indicate and they still need some time to improve it.

Google can hardly put a picture of a white male on their website. They're so deep in the swamp of virtue signalling it's a miracle they haven't yet drowned.

Google has many photos of white males on their website wtf: https://about.google/google-in-america/

It's by no means the whole story, but Google's very significant aversion to brand risk is definitely part of why they move slowly.

(Speaking as someone who's worked on launching several somewhat risky technologies at Google.)

Well not exactly. Not coming out until later when presumably GPT4 will have grown as much as well. So far each time, Google has failed to catch up to OpenAI. Hopefully they do however eventually.

> presumably GPT4 will have grown as much as well

Most of the comments I see on Hacker News claim ChatGPT is getting worse at different things (though I don't believe those claims).

Even if they're perpetually a year behind, that's a strong competitive spur to keep OpenAI on the move.

I’m afraid it won’t be nearly as good as GPT4, because of how lax Open AI can be with intellectual property. Google will not be able to train their model on Libgen or Opensubtitles, because they can’t afford the risk.

won't be available for regular devs until probably Q2 next year, OpenAI will have probably released GPT5 or whatever new model by then. And GPT4 was done training in 2022, the fact Google is thumping their chest about being 2 years behind a much smaller company is kind of pathetic

That's actually not a bad achievement considering Google is now the new IBM.