Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal

Re:Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal by squimmy26 on Saturday December 20, 2025 @10:55AM (#46337028)

How certain can we be that these improvements aren't just a result of Gemini 3 Pro pre-training on endless internet writeups of where 2.5 has struggled (and almost certainly what a human would have done instead)?

In other words, how much of this improvement is true generalization vs memorization?

[ Parent | on HN ]

Re:Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal by zurfer on Saturday December 20, 2025 @11:33AM (#46337345)

You're too kind. Even the CEO of Google retweeted how well Gemini 2.5 did on Pokemon. There is a high chance that now it's explicitly part of the training regime. We kind of need a different kind of game to know how well it generalizes.

[ Parent | on HN ]

Re:Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal by kqr on Saturday December 20, 2025 @01:25PM (#46338306)

I have a draft doing this with text adventures: https://entropicthoughts.com/updated-llm-benchmark

[ Parent | on HN ]

Re:Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal by prmoustache on Saturday December 20, 2025 @02:31PM (#46338845)

Isn't that the point of a new model anyway?

[ Parent | on HN ]

Re:Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal by DANmode on Sunday December 21, 2025 @01:10AM (#46342670)

Yes. Sort of.

Just don’t confuse it with a random benchmark!

[ Parent | on HN ]

Re:Gemini 3 Pro vs. 2.5 Pro in Pokemon Crystal by MrCheeze on Sunday December 21, 2025 @03:36AM (#46343218)

There were no such writeups, 99% of the discussion about difficulties in Crystal were in twitch and discord chats where Google doesn't scrape. (It hadn't yet gotten the public attention that Claude and Gemini's runs of Pokemon Red and Blue have gotten.)

That said, this writeup itself will probably be scraped and influence Gemini 4.

[ Parent | on HN ]