And a lot of human competitions aren't designed in such a way that the competition even makes sense with "AI." A lot of video games make this pretty obvious. It's relatively simple to build an aimbot in a first-person shooter that can outperform the most skilled humans. Even in ostensibly strategic games like Starcraft, bots can micro in ways that are blatantly impossible for humans and which don't really feel like an impressive display of Starcraft skill.
Another great example was IBM Watson playing Jeopardy! back in 2011. We were supposed to be impressed with Watson's natural language capabilities, but if you know anything about high-level Jeopardy! then you know that all you were really seeing is that robots have better reflexes than humans, which is hardly impressive.
Sure I do agree that the web search is too far, because it's literally cheating. But stockfish is super human at chess, it doesn't really matter that it can do this by leveraging the strengths of a computer.
It's like if you were building an AI robot to run a marathon against a human opponent, except you let the AI robot ride a motorcycle and force the human to stay on foot.
Did you read the article? It's clearly shown that with or without search it doesn't make much of a difference how good it actually is.
The differences even among humans between the absolute best & those out of the top 10 tend to be pretty drastic. And a non-IM against Magnus won't even understand what's going on. You could similarly claim that Magnus just memorized a bunch of openings which is similar to criticism GMs level too which is why Chess960 is now gaining more traction. My point is that there's not really such a thing as "fair" in a competition.
Re geoguessr, why not let them whatever tools are available? I have similar critiques about bike racing & restrictions on the technology advancements they can put on the bike. But every competition chooses arbitrary lines to draw which compose the rules so it doesn't really matter.
On the other hand you could generally beat the first "1v1 mid" bot by just cutting the wave behind its tower. So adaptation to new stuff was not good in isolation.
I would have loved to know whether given more time/prep/replays/practice pros would have figured out the holes. My guess is yes
The only difference might be compression, model weights throw away the noise and save the signal only.
For me humans versus machines is not an interesting competition. Machines will always win in a narrow specialized domain.
A more interesting competition is a very experienced human, versus an amateur who knows how to use A.I. Statistical/probabilistic models get confused, and they can easily wander aimlessly into rabbit holes. But a human who knows how to control the A.I. but is amateur at that particular narrow task, could guide it and at the end perform the same, or even better than the more experienced person.
In chess that's not true due to the super narrowed down domain of 64 squares and 6 different pieces, but anything that is more general a natty intelligence is necessary.
When I use it for programming, I never ask it to write code, i guide it to write that function, and use that other function from a library. If it is let free to guess, it will guess correctly 90% to 99% of the time, but if it is instructed then the code is almost flawless, nine nines percent of accuracy.
So 98% of the moves a computer will play are not especially surprising at all. A strong human will just about always have at least considered the move and even if not - they'll immediately understand the point. And in the 2% there's a relatively simple explanation. Computer's inability to make short-term mistakes lets them consider ideas humans never would. For instance humans tend to like material, yet there are a shockingly large number of positions where a modern computer will sac a piece and then just continue on playing a piece down in what "feels" like a fairly normal position. It simply turns out that your opponent simply has no way to convert their material advantage, and so your positional advantage will tell in the longrun, even being a piece down! At least if you're a computer...
This has led to some interesting outcomes. For instance Fabiano Caruana, a top 10 player in the world, is extremely well known for his exceptional level of deep and creative opening preparation, all computer approved. But in more than a few instances he's ended up in positions that look bad but where a computer will say he's practically winning, and ultimately go on to lose the game. It's simply because these sort of positions might indeed be objectively winning, but it may require 10 or 20 practically perfect moves - whereas a single subtle mistake means you lose. And it's extremely hard for even the best players in the world to play like this.
How are these points connected? Playing blindfolded doesn’t require being able to calculate 30 moves deep (or any particular number).
Being able to remember/visualize an N move sequence without losing the thread while blindfolded is not at all the same thing as being able to calculate N moves deep.
Another example to illustrate the point is the ICCF (International Correspondence Chess Federation). Were computers comparably competent at long-term play as they are at short-term, then there wouldn't even be a competition. It'd simply be who has the strongest computer. But in reality that seems to play no particularly decisive factor. For instance, as in "normal" chess, there remains a huge gender divide in ratings, yet females certainly have no less access to competent hardware than males.
Chess, for a stronger player, is very much like a language - in fact it uses the exact same area of the brain. It's like when you read these words, you're not consciously thinking at all - the meaning just comes to you immediately. And in fact you could trivially recall everything I said (even if not necessarily verbatim) if you just thought for a second or so. But simultaneously it's not like you actually made any effort whatsoever to memorize it.
So how long of a conversation could you hold with yourself in your mind? Practically endless, and you could probably reconstruct the overwhelming majority of it on demand. It's the same with chess. Meandering around in your mind to positions of an arbitrary depth is not difficult for a strong player. And the person I was responding to felt that a player writing down some analysis to move 20 would be some meaningful form of cheating. In reality, I'd absolutely love for my opponent to be able to write down their analyses. It'd waste just a monumental amount of time and afford no advantage whatsoever. It'd be akin to you writing down the conversation from your mind.
It's not interesting playing chess against Stockfish 17, even for high-level GMs. It's alien and just crushes every human. Writing down an analysis to 20 move depth, following some lines to 30 or more, would be cheating for humans. It would take way too long (exceeding any time controls and more importantly exceeding the lifetime of the human), a powerful computer can just crunch it in seconds. Referencing a tablebase of endgames for 7 pieces would also be cheating, memorizing 7 terabytes of bitwise layouts is absurd but the computer just stores that on its hard drive.
Human geoguessr players have impressive memories way above baseline with respect to regional infrastructure, geography, trees, road signs, written language, and other details. Likewise, human Jeopardy players know an awful lot of trivia. Once you get to something like Scrabble or chess, it's less and less about knowing words or knowing moves, but more about synthesizing that knowledge intelligently.
One would expect a human to recognize some domain names like, I don't know, osu.edu: lots of people know that's Ohio State University, one of the biggest schools in the US, located in Columbus, Ohio. They don't have to cheat and go to an external resource. One would expect a human (a top human player, at least) to know that taxilinder.at is based in Austria. One would never expect any human to have every business or domain name memorized.
With modern AI models trained on internet data, searching the internet is not that different from querying its own training data.