The AI should be forced to use the same rules as the human. Not the other way around. The AI shouldn't be using outside resources.

Another rule bans "using third-party software or scripts in order to gain an unfair advantage over other players."

So is it even possible for O3 to beat another player while complying with the rules?

If a player uses such a model, the model is third-party and the player is cheating.

But: when a specific model is itself under test, I would say that during the test it becomes "first" (or second?) party rather than "third".

I think third-party here means not produced by GeoGuesser rather than not produced by the player.

I think that's part of the point they're making, hence "They should've simply redone the test."

An AI being better than a human at doing a google search and then skimming a bunch of pages to find location-related terms isn't as interesting of a result.

How the heck is not? Computers are looking into screenshots and searching the internet to support their "thinking", that's amazing! Have we become so used to AI and what was impossible 6 months ago is shruggable today?

I've being doing this MIND-Dash diet lately and it's amazing I can just take a picture of whatever (nutritional info / ingredients are perfect for that) and just ask if it fits my plan and it tells me back into what bucket it falls, with detailed breakdown of macros in support of some additional goals I have (muscle building for powerlifting). It's amazing! And it does in 2 minutes passively what it would take me 5-10 active search.

Yeah, it's a funny take because this is in fact a more advanced form of AI with autonomous tool use that is just now emerging in 2025. You might say "They could search the web in 2024 too" but that wasn't autonomous on its own, but required telling so or checking a box. This one is piecing ideas together like "Wait, I should Google for this" and that is specifically a new feature for OpenAI o3 that wasn't even in o1.

While it isn't entirely in the spirit of GeoGuesser, it is a good test of the capabilities where being great at GeoGuesser in fact becomes the lesser news here. It will still be if disabling this feature.

I fully expect that someday the news will announce, "The AI appears to be dismantling the moons of Jupiter and turning them into dense, exotic computational devices which it is launching into low solar orbit. We're not sure why. The AI refused to comment."

And someone will post, "Yeah, but that's just computer aideded design and manufacturing. It's not real AI."

The first rule of AI is that the goalposts always move. If a computer can do it, by definition, it isn't "real" AI. This will presumably continue to apply even as the Terminator kicks in the front door.

Yes, but I choose to interpret that as a good thing. It is good that progress is so swift and steady that we can afford to keep moving the goalposts.

Take cars as a random example: progress there isn't fast enough that we keep moving the goalposts for eg fuel economy. (At least not nearly as much.) A car with great fuel economy 20 years ago is today considered at least still good in terms of fuel economy.

And if you account for the makeup of the fleet on the road overall, a great fuel economy car from 1995 (say, a Prizm), still beats the median vehicle on the road, which is certainly an SUV weighing twice as much and gets worse mileage.

In the same way a calculator performing arithmetic faster than humans isn't impressive. The same way running regex over a million lines and the computer beating a human in search isn't impressive

Neither is impressive solely because we've gotten used to them. Both were mind-blowing back in the day.

When it comes to AI - and LLMs in particular - there’s a large cohort of people who seem determined to jump straight from "impossible and will never happen in our lifetime" to "obvious and not impressive", without leaving any time to actually be impressed by the technological achievement. I find that pretty baffling.

I agree, but without removing search you cannot decouple. Has it embedded a regex method and is just leveraging that? Or is it doing something more? Yes, even the regex is still impressive but it is less impressive that doing something more complicated and understanding context and more depth.

I think both are very impressive, world shattering capabilities. Just because they have become normalized doesn't make it any less impressive in my view.

That's a fair point, and I would even agree. Though I think we could agree that it is fair to interpret "impressive" in this context as "surprising". There's lots of really unsurprising things that are incredibly impressive. But I think the general usage of the word here is more akin to surprisal.

That isn't what's happening though. I re-ran those two rounds, this time without search, and it changed nothing. I updated the post with details, you can verify it yourself.

Claiming the AI is just using Google is false and dismissing a truly incredible capability.

But then they couldn't make a click bait title for the article.